Original post
\nI recently managed to get all my pre- and postprocessing through the scoring pipeline (Tensorflow). Although I won't share my code here at this point, I will share a few things, as well as what I have learned from this experience:
\nI started from the message by @mdecoster that there were probably empty samples in the test set (or maybe invalid parquet files or entries that result in empty samples).
\nThe first step that allowed me to make progress was to add a custom \"CatchEmpty\" layer at the very input of my model (right after the remove nan's line that is also in my dummy submission). This detects empty input samples and replaces them with a single-timestep dummy zeros frame. So the code roughly becomes (loaded_model is the model I trained, saved to disk and reloaded in my TfLite creation notebook, CustomPreprocessing() and CustomPostprocessing() are placeholders for custom layers I wrote):
inputs = tf.keras.Input(shape=(NUM_FEATURES,), dtype=tf.float32, name=)\n x = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n x = CatchEmpty()(x) \n x = CustomPreprocessing()(x) \n x = loaded_model(x)\n x = CustomPostprocessing()(x) \n out = tf.keras.layers.Activation(, name=)(x_out)\n inference_model = tf.keras.Model(inputs=inputs, outputs=out)\nStarting from the information from @sohier that many of the submission failures were due to dimension mismatches and from my earlier insight that indeed some dimension mismatches did not cause errors or warnings in my local notebook, I started stepping through each individual line of my pre- and postprocessing code and printing out the tensors and tensor shapes for three situations:
\n(1) multiple \"normal\" train set frames as inputs (selected to mimic specific cases)
\n(2) a single \"normal\" train set frame as inputs (selected to mimic specific cases)
\n(3) an empty input generated as tf.zeros([0, NUM_FEATURES], dtype = tf.dtypes.float32)
\n(4) an all-zeros input tf.zeros([1, NUM_FEATURES], dtype = tf.dtypes.float32)
\nThis led to a number of corrections in my code, again making sure to catch a number of possible very specific situations.
After making sure in this way that all standard tf functions (as well as my model) received tensors with their expected input shapes, I found that all functions I used in the previous ASL-ISLR competition did in fact work. For example, my current successful code uses things like where, reduce_sum, not_equal, gather, cond, ones_like, zeros_like, etc.
Note that this does not guarantee that all functionalities in conversions from PyTorch also work (I'm quite happy with using Tensorflow).
\nUpdate:
\nOut of curiosity I did a few more tests
\n(1) removed everything from my CatchEmpty() layer except x = x[None] -> PASSED scoring (??)
\n(2) moved x = x[None] back to where it originally was -- after CustomPreprocessing() -- and changed the code in CustomPreprocessing() from working with 3D tensors back to working with 2D tensors (without changing the operators and with careful checking of all dimensions and corner cases) -> FAILED scoring (??)
So now I'm not sure anymore that mere empty samples of the form tf.zeros([0, NUM_FEATURES], dtype = tf.dtypes.float32) are the corner case that causes the scoring failures. However, having x = x[None] in front seems to be crucial for getting things to work , but since I'm not sure exactly what this does behind the screen, I have no clue what we're dealing with …
Original post
\nI recently managed to get all my pre- and postprocessing through the scoring pipeline (Tensorflow). Although I won't share my code here at this point, I will share a few things, as well as what I have learned from this experience:
\nI started from the message by @mdecoster that there were probably empty samples in the test set (or maybe invalid parquet files or entries that result in empty samples).
\nThe first step that allowed me to make progress was to add a custom \"CatchEmpty\" layer at the very input of my model (right after the remove nan's line that is also in my dummy submission). This detects empty input samples and replaces them with a single-timestep dummy zeros frame. So the code roughly becomes (loaded_model is the model I trained, saved to disk and reloaded in my TfLite creation notebook, CustomPreprocessing() and CustomPostprocessing() are placeholders for custom layers I wrote):
inputs = tf.keras.Input(shape=(NUM_FEATURES,), dtype=tf.float32, name=)\n x = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n x = CatchEmpty()(x) \n x = CustomPreprocessing()(x) \n x = loaded_model(x)\n x = CustomPostprocessing()(x) \n out = tf.keras.layers.Activation(, name=)(x_out)\n inference_model = tf.keras.Model(inputs=inputs, outputs=out)\nStarting from the information from @sohier that many of the submission failures were due to dimension mismatches and from my earlier insight that indeed some dimension mismatches did not cause errors or warnings in my local notebook, I started stepping through each individual line of my pre- and postprocessing code and printing out the tensors and tensor shapes for three situations:
\n(1) multiple \"normal\" train set frames as inputs (selected to mimic specific cases)
\n(2) a single \"normal\" train set frame as inputs (selected to mimic specific cases)
\n(3) an empty input generated as tf.zeros([0, NUM_FEATURES], dtype = tf.dtypes.float32)
\n(4) an all-zeros input tf.zeros([1, NUM_FEATURES], dtype = tf.dtypes.float32)
\nThis led to a number of corrections in my code, again making sure to catch a number of possible very specific situations.
After making sure in this way that all standard tf functions (as well as my model) received tensors with their expected input shapes, I found that all functions I used in the previous ASL-ISLR competition did in fact work. For example, my current successful code uses things like where, reduce_sum, not_equal, gather, cond, ones_like, zeros_like, etc.
Note that this does not guarantee that all functionalities in conversions from PyTorch also work (I'm quite happy with using Tensorflow).
\nUpdate:
\nOut of curiosity I did a few more tests
\n(1) removed everything from my CatchEmpty() layer except x = x[None] -> PASSED scoring (??)
\n(2) moved x = x[None] back to where it originally was -- after CustomPreprocessing() -- and changed the code in CustomPreprocessing() from working with 3D tensors back to working with 2D tensors (without changing the operators and with careful checking of all dimensions and corner cases) -> FAILED scoring (??)
So now I'm not sure anymore that mere empty samples of the form tf.zeros([0, NUM_FEATURES], dtype = tf.dtypes.float32) are the corner case that causes the scoring failures. However, having x = x[None] in front seems to be crucial for getting things to work , but since I'm not sure exactly what this does behind the screen, I have no clue what we're dealing with …
Thank you. I successfully made preprocessing into submission. x = x[None] at the starting of preprocessing certainly worked.
", "rawMarkdown": "Thank you. I successfully made preprocessing into submission. x = x[None] at the starting of preprocessing certainly worked.", "votes": 4 }, { "id": 2311786, "postDate": "2023-06-21T13:01:28.060Z", "content": "I just changed my preprocessing function. The old version didn't include any indexing operations, and x=x[None]was placed after this.
Now, I have added some index operations, and they cause an error. However, placing x=x[None] before these operations helped.
I believe the main issue lies in the indexing operations.
\nI am thankful for this post as a reminder. I think this post should be pinned.
", "rawMarkdown": "I just changed my preprocessing function. The old version didn't include any indexing operations, and `x=x[None] `was placed after this.\n\nNow, I have added some index operations, and they cause an error. However, placing `x=x[None]` before these operations helped.\n\nI believe the main issue lies in the indexing operations.\n\nI am thankful for this post as a reminder. I think this post should be pinned.", "votes": 1 }, { "id": 2292716, "postDate": "2023-06-08T15:38:54.027Z", "content": "I couldn't help but applaud when I saw that the scoring process started and didn't fail even after three minutes.
", "rawMarkdown": "I couldn't help but applaud when I saw that the scoring process started and didn't fail even after three minutes.", "votes": 1 }, { "id": 2291098, "postDate": "2023-06-07T10:22:36.143Z", "content": "Thanks! This is really good investigative work. Will give it a try :)
\nEDIT: It worked! My model was failing on empty inputs - once this was fixed, and after a few attempts to work around bugs in the TFLite converter, I was able to get it working. Thanks a lot!
", "rawMarkdown": "Thanks! This is really good investigative work. Will give it a try :)\n\nEDIT: It worked! My model was failing on empty inputs - once this was fixed, and after a few attempts to work around bugs in the TFLite converter, I was able to get it working. Thanks a lot!", "votes": 2 }, { "id": 2313976, "postDate": "2023-06-23T05:07:12.457Z", "content": "Thanks for your sharing, it help me a lot!
", "rawMarkdown": "Thanks for your sharing, it help me a lot!" } ], "comments": [ { "id": 2290730, "author_name": "Rohith Ingilela", "author_url": "", "post_date": "2023-06-07T04:22:00.513000", "content": "Thank you. I successfully made preprocessing into submission. x = x[None] at the starting of preprocessing certainly worked.
", "votes": 4, "replies": [] }, { "id": 2311786, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-06-21T13:01:28.060000", "content": "I just changed my preprocessing function. The old version didn't include any indexing operations, and x=x[None]was placed after this.
Now, I have added some index operations, and they cause an error. However, placing x=x[None] before these operations helped.
I believe the main issue lies in the indexing operations.
\nI am thankful for this post as a reminder. I think this post should be pinned.
", "votes": 1, "replies": [] }, { "id": 2292716, "author_name": "canlion", "author_url": "", "post_date": "2023-06-08T15:38:54.027000", "content": "I couldn't help but applaud when I saw that the scoring process started and didn't fail even after three minutes.
", "votes": 1, "replies": [] }, { "id": 2291098, "author_name": "anokas", "author_url": "", "post_date": "2023-06-07T10:22:36.143000", "content": "Thanks! This is really good investigative work. Will give it a try :)
\nEDIT: It worked! My model was failing on empty inputs - once this was fixed, and after a few attempts to work around bugs in the TFLite converter, I was able to get it working. Thanks a lot!
", "votes": 2, "replies": [] }, { "id": 2313976, "author_name": "treeleaves30760", "author_url": "", "post_date": "2023-06-23T05:07:12.457000", "content": "Thanks for your sharing, it help me a lot!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2289537": "---------------------------------------\n**Original post**\n\n---------------------------------------\n\nI recently managed to get all my pre- and postprocessing through the scoring pipeline (Tensorflow). Although I won't share my code here at this point, I will share a few things, as well as what I have learned from this experience:\n\nI started from the message by @mdecoster that there were probably empty samples in the test set (or maybe invalid parquet files or entries that result in empty samples). \nThe first step that allowed me to make progress was to add a custom \"CatchEmpty\" layer at the very input of my model (right after the remove nan's line that is also in my dummy submission). This detects empty input samples and replaces them with a single-timestep dummy zeros frame. So the code roughly becomes (```loaded_model``` is the model I trained, saved to disk and reloaded in my TfLite creation notebook, ```CustomPreprocessing()``` and ```CustomPostprocessing()``` are placeholders for custom layers I wrote):\n\n``` python\n inputs = tf.keras.Input(shape=(NUM_FEATURES,), dtype=tf.float32, name=\"inputs\")\n x = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n x = CatchEmpty()(x) # this layer starts with the dimension extension x=x[None] \n x = CustomPreprocessing()(x) \n x = loaded_model(x)\n x = CustomPostprocessing()(x) # this also contains the dimension reduction x=x[0,:,:] \n out = tf.keras.layers.Activation(\"linear\", name=\"outputs\")(x_out)\n inference_model = tf.keras.Model(inputs=inputs, outputs=out)\n```\n\nStarting from the information from @sohier that many of the submission failures were due to dimension mismatches and from my earlier insight that indeed some dimension mismatches did not cause errors or warnings in my local notebook, I started stepping through each individual line of my pre- and postprocessing code and printing out the tensors and tensor shapes for three situations: \n(1) multiple \"normal\" train set frames as ```inputs``` (selected to mimic specific cases)\n(2) a single \"normal\" train set frame as ```inputs``` (selected to mimic specific cases)\n(3) an empty input generated as ```tf.zeros([0, NUM_FEATURES], dtype = tf.dtypes.float32)```\n(4) an all-zeros input ```tf.zeros([1, NUM_FEATURES], dtype = tf.dtypes.float32)```\nThis led to a number of corrections in my code, again making sure to catch a number of possible very specific situations.\n\nAfter making sure in this way that all standard tf functions (as well as my model) received tensors with their expected input shapes, I found that all functions I used in the previous ASL-ISLR competition did in fact work. For example, my current successful code uses things like ```where```, ```reduce_sum```, ```not_equal```, ```gather```, ```cond```, ```ones_like```, ```zeros_like```, etc.\n\nNote that this does not guarantee that all functionalities in conversions from PyTorch also work (I'm quite happy with using Tensorflow).\n\n---------------------------------------\n**Update:** \n\n---------------------------------------\n\nOut of curiosity I did a few more tests\n\n(1) removed everything from my ```CatchEmpty()``` layer except ```x = x[None]``` -> **PASSED scoring** (??)\n(2) moved ```x = x[None]``` back to where it originally was -- after ```CustomPreprocessing()``` -- and changed the code in ```CustomPreprocessing()``` from working with 3D tensors back to working with 2D tensors (without changing the operators and with careful checking of all dimensions and corner cases) -> **FAILED scoring** (??)\n\nSo now I'm not sure anymore that mere empty samples of the form ```tf.zeros([0, NUM_FEATURES], dtype = tf.dtypes.float32)``` are the corner case that causes the scoring failures. However, **having** ```x = x[None]``` **in front seems to be crucial for getting things to work** , but since I'm not sure **exactly** what this does behind the screen, I have no clue what we're dealing with ...\n", "2290730": "Thank you. I successfully made preprocessing into submission. x = x[None] at the starting of preprocessing certainly worked.", "2311786": "I just changed my preprocessing function. The old version didn't include any indexing operations, and `x=x[None] `was placed after this.\n\nNow, I have added some index operations, and they cause an error. However, placing `x=x[None]` before these operations helped.\n\nI believe the main issue lies in the indexing operations.\n\nI am thankful for this post as a reminder. I think this post should be pinned.", "2292716": "I couldn't help but applaud when I saw that the scoring process started and didn't fail even after three minutes.", "2291098": "Thanks! This is really good investigative work. Will give it a try :)\n\nEDIT: It worked! My model was failing on empty inputs - once this was fixed, and after a few attempts to work around bugs in the TFLite converter, I was able to get it working. Thanks a lot!", "2313976": "Thanks for your sharing, it help me a lot!" } }