Hello fellow Kagglers,
\nAs many of you, I keep getting submission errors.
\nThe main cause are three basic preprocessing steps.
\nWithout proper preprocessing the challenge becomes to create a model that somehow performs well on raw input, rather than creating a high performing model on nicely preprocessed data.
Here is my code, all steps except the resizing fail during submission, but are successful in a Tensorflow Lite 2.9.1 environment.
\nIf we manage to get these basic preprocessing steps working in a submission we can all start to focus on engineering complex models!
\nN_COLS0 = 84\nN_COLS = 42\nN_TARGET_FRAMES N_TARGET_FRAMES = 128 # or any other value you like\n\n# Simple Tensorflow layer\nclass PreprocessLayer(tf.keras.layers.Layer):\n def __init__(self):\n super(PreprocessLayer, self).__init__()\n\n @tf.function(\n input_signature=(tf.TensorSpec(shape=[None,N_COLS0], dtype=tf.float32),),\n )\n def call(self, data0): \n # Fill NaN Values With 0\n data = tf.where(tf.math.is_nan(data0), 0.0, data0)\n1) Determine the dominant hand [FAILS]
\n# Compute summed coordinates\nleft_hand = tf.slice(data, [0,0], [-1,42])\nleft_hand_sum = tf.math.reduce_sum(left_hand)\nright_hand = tf.slice(data, [0,42], [-1,42])\nright_hand_sum = tf.math.reduce_sum(right_hand)\n# Dominant hand is determined based on larger sum of coordinates\nleft_dominant = left_hand_sum >= right_hand_sum\n\n# Slice dominant hand | N_FRAMESx84 -> N_FRAMESx42\nif left_dominant:\n data = tf.slice(data, [0,0], [-1,42])\nelse:\n data = tf.slice(data, [0,42], [-1,42])\n2) Filter out all frames without hand coordinates [FAILS]
\n# Get indices of frames with coordinates, NaN is filled with 0 thus will sum to 0 and filtered\nnon_empty_frames_idxs = tf.where(tf.math.reduce_sum(data, axis=[1]) > 0)\n\n# Filter data on frames with coordinates for hand\ndata = tf.gather(data, non_empty_frames_idxs, axis=0, name='gather_non_empty_frames_idxs')\n3) Pad input to target length [FAILS]
\n# Number of frames in video\nN_FRAMES = tf.shape(data0)[0]\n# Pad to number of target frames (128)\npadding = tf.maximum(0, N_TARGET_FRAMES - N_FRAMES)\ndata = tf.concat([data, tf.zeros([padding, N_COLS])], axis=0)\nnon_empty_frames_idxs = tf.concat([non_empty_frames_idxs, tf.fill([padding], -1.0)], axis=0)\n4) Resize To Target Size [SUCCEEDS]
\n# Downsample Video\ndata = tf.image.resize(\n data[:,:,tf.newaxis],\n [N_TARGET_FRAMES, N_COLS],\n method=tf.image.ResizeMethod.BILINEAR,\n antialias=False,\n)\ndata = tf.squeeze(data, axis=[2])\n\n# Resize Non Empty Frame Indices\nnon_empty_frames_idxs = tf.image.resize(\n non_empty_frames_idxs[:,tf.newaxis,tf.newaxis],\n [N_TARGET_FRAMES, 1],\n method=tf.image.ResizeMethod.BILINEAR,\n antialias=False,\n)\nnon_empty_frames_idxs = tf.squeeze(non_empty_frames_idxs, axis=[1,2])\nI was wondering if you:
\nHello fellow Kagglers,
\nAs many of you, I keep getting submission errors.
\nThe main cause are three basic preprocessing steps.
\nWithout proper preprocessing the challenge becomes to create a model that somehow performs well on raw input, rather than creating a high performing model on nicely preprocessed data.
Here is my code, all steps except the resizing fail during submission, but are successful in a Tensorflow Lite 2.9.1 environment.
\nIf we manage to get these basic preprocessing steps working in a submission we can all start to focus on engineering complex models!
\nN_COLS0 = 84\nN_COLS = 42\nN_TARGET_FRAMES N_TARGET_FRAMES = 128 # or any other value you like\n\n# Simple Tensorflow layer\nclass PreprocessLayer(tf.keras.layers.Layer):\n def __init__(self):\n super(PreprocessLayer, self).__init__()\n\n @tf.function(\n input_signature=(tf.TensorSpec(shape=[None,N_COLS0], dtype=tf.float32),),\n )\n def call(self, data0): \n # Fill NaN Values With 0\n data = tf.where(tf.math.is_nan(data0), 0.0, data0)\n1) Determine the dominant hand [FAILS]
\n# Compute summed coordinates\nleft_hand = tf.slice(data, [0,0], [-1,42])\nleft_hand_sum = tf.math.reduce_sum(left_hand)\nright_hand = tf.slice(data, [0,42], [-1,42])\nright_hand_sum = tf.math.reduce_sum(right_hand)\n# Dominant hand is determined based on larger sum of coordinates\nleft_dominant = left_hand_sum >= right_hand_sum\n\n# Slice dominant hand | N_FRAMESx84 -> N_FRAMESx42\nif left_dominant:\n data = tf.slice(data, [0,0], [-1,42])\nelse:\n data = tf.slice(data, [0,42], [-1,42])\n2) Filter out all frames without hand coordinates [FAILS]
\n# Get indices of frames with coordinates, NaN is filled with 0 thus will sum to 0 and filtered\nnon_empty_frames_idxs = tf.where(tf.math.reduce_sum(data, axis=[1]) > 0)\n\n# Filter data on frames with coordinates for hand\ndata = tf.gather(data, non_empty_frames_idxs, axis=0, name='gather_non_empty_frames_idxs')\n3) Pad input to target length [FAILS]
\n# Number of frames in video\nN_FRAMES = tf.shape(data0)[0]\n# Pad to number of target frames (128)\npadding = tf.maximum(0, N_TARGET_FRAMES - N_FRAMES)\ndata = tf.concat([data, tf.zeros([padding, N_COLS])], axis=0)\nnon_empty_frames_idxs = tf.concat([non_empty_frames_idxs, tf.fill([padding], -1.0)], axis=0)\n4) Resize To Target Size [SUCCEEDS]
\n# Downsample Video\ndata = tf.image.resize(\n data[:,:,tf.newaxis],\n [N_TARGET_FRAMES, N_COLS],\n method=tf.image.ResizeMethod.BILINEAR,\n antialias=False,\n)\ndata = tf.squeeze(data, axis=[2])\n\n# Resize Non Empty Frame Indices\nnon_empty_frames_idxs = tf.image.resize(\n non_empty_frames_idxs[:,tf.newaxis,tf.newaxis],\n [N_TARGET_FRAMES, 1],\n method=tf.image.ResizeMethod.BILINEAR,\n antialias=False,\n)\nnon_empty_frames_idxs = tf.squeeze(non_empty_frames_idxs, axis=[1,2])\nI was wondering if you:
\nHave posted here
\nRegarding NaN frames or hands, using a preprocessing layer for selected columns that included frame -
xlh = tf.slice(x0, [0,1], [-1,42 ]) # lh slice x y (n.b. 0 is frame so use 1)
\nxrh = tf.slice(x0, [0,43], [-1,42 ]) # rh slice x y
\nxrhv = tf.boolean_mask(xrh, tf.reduce_all(~tf.math.is_nan(xrh), axis=1)) # remove NaN rows
\nxlhv =tf.boolean_mask(xlh, tf.reduce_all(~tf.math.is_nan(xlh), axis=1))
\nx = tf.cond(tf.math.is_nan(tf.math.zero_fraction(xlhv)), lambda: tf.identity(xrhv), lambda: tf.identity(xlhv))
Can confirm this works in submission without errors. Selected columns were frame and x left y left x right y right so all left are together and all right are together.
", "rawMarkdown": "Have posted [here](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/413598#2280786)\nRegarding NaN frames or hands, using a preprocessing layer for selected columns that included frame -\n\n`xlh = tf.slice(x0, [0,1], [-1,42 ]) # lh slice x y (n.b. 0 is frame so use 1)`\n`xrh = tf.slice(x0, [0,43], [-1,42 ]) # rh slice x y`\n`xrhv = tf.boolean_mask(xrh, tf.reduce_all(~tf.math.is_nan(xrh), axis=1)) # remove NaN rows`\n`xlhv =tf.boolean_mask(xlh, tf.reduce_all(~tf.math.is_nan(xlh), axis=1))`\n`x = tf.cond(tf.math.is_nan(tf.math.zero_fraction(xlhv)), lambda: tf.identity(xrhv), lambda: tf.identity(xlhv))`\n\nCan confirm this works in submission without errors. Selected columns were frame and x left y left x right y right so all left are together and all right are together. ", "votes": 1 }, { "id": 2286262, "postDate": "2023-06-03T10:41:18.387Z", "content": "\nI've spent some time trying to find edge cases with your code, so far I've found two with NaN removal and missing frame removal: see https://www.kaggle.com/mdecoster/edge-cases-and-local-inference
\nNote that this notebook still does not pass submission (despite working locally on all files), so other failure cases still remain.
", "rawMarkdown": "Hi @markwijkhuizen \n\nI've spent some time trying to find edge cases with your code, so far I've found two with NaN removal and missing frame removal: see https://www.kaggle.com/mdecoster/edge-cases-and-local-inference\n\nNote that this notebook still does not pass submission (despite working locally on *all* files), so other failure cases still remain.", "replies": [ { "id": 2286421, "postDate": "2023-06-03T12:48:55.377Z", "content": "Hi @mdecoster
\nGreat work on detecting the edge cases, this makes the preprocessing more robust.
\nI suspect the failing submission to be caused by the environment, if even your inference loop with all the edge case guards fails.
\nKeep in mind the preprocessing code provided does run correctly on the full training dataset as well.
Can you show the error comments?
", "rawMarkdown": "Can you show the error comments?", "replies": [ { "id": 2286179, "postDate": "2023-06-03T09:32:24.320Z", "content": "The submission fails with the message: Submission Scoring Error
Let me run the code.
", "rawMarkdown": "Let me run the code.\n" } ] }, { "id": 2293634, "postDate": "2023-06-09T11:09:41.473Z", "content": "The Submission Scoring Error is most probably because your model output does not match with the expected output. There are information about how the evaluation is done. Make sure the code snippet provided in the evaluation section runs successfully with your model.
\nI had the same problem. This line np.argmax(output[REQUIRED_OUTPUT], axis=1) in the evaluation code snippet is expecting the model output in 2D (batch_size, one_hot encoded data). After adjusting the tflite model output, everything worked perfect and I got score of zero. Not bad huh?
Have posted here
\nRegarding NaN frames or hands, using a preprocessing layer for selected columns that included frame -
xlh = tf.slice(x0, [0,1], [-1,42 ]) # lh slice x y (n.b. 0 is frame so use 1)
\nxrh = tf.slice(x0, [0,43], [-1,42 ]) # rh slice x y
\nxrhv = tf.boolean_mask(xrh, tf.reduce_all(~tf.math.is_nan(xrh), axis=1)) # remove NaN rows
\nxlhv =tf.boolean_mask(xlh, tf.reduce_all(~tf.math.is_nan(xlh), axis=1))
\nx = tf.cond(tf.math.is_nan(tf.math.zero_fraction(xlhv)), lambda: tf.identity(xrhv), lambda: tf.identity(xlhv))
Can confirm this works in submission without errors. Selected columns were frame and x left y left x right y right so all left are together and all right are together.
", "votes": 1, "replies": [] }, { "id": 2286262, "author_name": "Mathieu De Coster", "author_url": "", "post_date": "2023-06-03T10:41:18.387000", "content": "\nI've spent some time trying to find edge cases with your code, so far I've found two with NaN removal and missing frame removal: see https://www.kaggle.com/mdecoster/edge-cases-and-local-inference
\nNote that this notebook still does not pass submission (despite working locally on all files), so other failure cases still remain.
", "votes": 0, "replies": [ { "id": 2286421, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2023-06-03T12:48:55.377000", "content": "Hi @mdecoster
\nGreat work on detecting the edge cases, this makes the preprocessing more robust.
\nI suspect the failing submission to be caused by the environment, if even your inference loop with all the edge case guards fails.
\nKeep in mind the preprocessing code provided does run correctly on the full training dataset as well.
Can you show the error comments?
", "votes": 0, "replies": [ { "id": 2286179, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2023-06-03T09:32:24.320000", "content": "The submission fails with the message: Submission Scoring Error
Let me run the code.
", "votes": 0, "replies": [] } ] }, { "id": 2293634, "author_name": "Salman Ahmad", "author_url": "", "post_date": "2023-06-09T11:09:41.473000", "content": "The Submission Scoring Error is most probably because your model output does not match with the expected output. There are information about how the evaluation is done. Make sure the code snippet provided in the evaluation section runs successfully with your model.
\nI had the same problem. This line np.argmax(output[REQUIRED_OUTPUT], axis=1) in the evaluation code snippet is expecting the model output in 2D (batch_size, one_hot encoded data). After adjusting the tflite model output, everything worked perfect and I got score of zero. Not bad huh?