I have a model that keeps failing after 10 to 20 minutes after submitting.
\nMy assumption is that it crashes on some sort of quirk on the dataset.
\nI have successfully tan the tflite engine on all available datasets.
\ni have also successfully ran it against dummy tensors [0, N], [1,N], … [99999, N].
\nDoes anyone have any insights of what else ai could try doing?
\nThank you all 🙏
Some tips to prevent failing submissions
\n1) Always start your preprocessing layer with
\n data0[None]\n2) Enforce correct shapes
\n\n = len(\n > :\n \n < :\n \n3) Fill NaN Values
\n\n\nFor debugging purposes, you could try if you submission succeeds when only these 3 preprocessing steps are applied.
", "rawMarkdown": "Some tips to prevent failing submissions\n\n1) Always start your preprocessing layer with\n```\ndata = data0[None]\n```\n\n2) Enforce correct shapes\n```\n# Crop/Pad\nN_FRAMES = len(data)\nif N_FRAMES > N_TARGET_FRAMES:\n data = tf.slice(data, [0,0], [N_TARGET_FRAMES, N_COLS])\nelif N_FRAMES < N_TARGET_FRAMES:\n data = tf.pad(data, [[0,N_TARGET_FRAMES-N_FRAMES], [0,0]])\n```\n\n3) Fill NaN Values\n```\n# Fill NaN\ndata = tf.where(tf.math.is_nan(data), tf.constant(0.0, data.dtype), data)\n```\n\nFor debugging purposes, you could try if you submission succeeds when only these 3 preprocessing steps are applied.", "votes": 1, "replies": [ { "id": 2363199, "postDate": "2023-07-28T15:00:20.527Z", "content": "Thanks for the reply. I think, in my case, the problem was some strange version incompatibility. I was exporting the model on my machine, and although it would load and run the whole dataset inside Kaggle, the sub would still fail. I am not saving the weights for the TF model and building/loading/exporting everything on Kaggle now.
\nIt is bizarre but seems to be working.
\nOut of curiosity, #2 is a pseudo-code or actual code? I ask because how would the TFLite engine be able to trace this if-else conditional inside the model?
", "rawMarkdown": "Thanks for the reply. I think, in my case, the problem was some strange version incompatibility. I was exporting the model on my machine, and although it would load and run the whole dataset inside Kaggle, the sub would still fail. I am not saving the weights for the TF model and building/loading/exporting everything on Kaggle now. \n\nIt is bizarre but seems to be working.\n\nOut of curiosity, #2 is a pseudo-code or actual code? I ask because how would the TFLite engine be able to trace this if-else conditional inside the model?", "replies": [ { "id": 2363206, "postDate": "2023-07-28T15:03:10.227Z", "content": "Happy you found the solution, correct TFLite conversion is quite a challenge in this competition…
\nThe code snippets are actual code snippets from a notebook with a succeeding submission.
", "rawMarkdown": "Happy you found the solution, correct TFLite conversion is quite a challenge in this competition...\n\nThe code snippets are actual code snippets from a notebook with a succeeding submission.", "votes": 2 } ] } ] }, { "id": 2362265, "postDate": "2023-07-28T00:41:24.227Z", "content": "I have a model that keeps failing after 10 to 20 minutes after submitting.
\nMy assumption is that it crashes on some sort of quirk on the dataset.
\nI have successfully tan the tflite engine on all available datasets.
\ni have also successfully ran it against dummy tensors [0, N], [1,N], … [99999, N].
\nDoes anyone have any insights of what else ai could try doing?
\nThank you all 🙏
Some tips to prevent failing submissions
\n1) Always start your preprocessing layer with
\n data0[None]\n2) Enforce correct shapes
\n\n = len(\n > :\n \n < :\n \n3) Fill NaN Values
\n\n\nFor debugging purposes, you could try if you submission succeeds when only these 3 preprocessing steps are applied.
", "votes": 1, "replies": [ { "id": 2363199, "author_name": "Adriano Passos", "author_url": "", "post_date": "2023-07-28T15:00:20.527000", "content": "Thanks for the reply. I think, in my case, the problem was some strange version incompatibility. I was exporting the model on my machine, and although it would load and run the whole dataset inside Kaggle, the sub would still fail. I am not saving the weights for the TF model and building/loading/exporting everything on Kaggle now.
\nIt is bizarre but seems to be working.
\nOut of curiosity, #2 is a pseudo-code or actual code? I ask because how would the TFLite engine be able to trace this if-else conditional inside the model?
", "votes": 0, "replies": [ { "id": 2363206, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2023-07-28T15:03:10.227000", "content": "Happy you found the solution, correct TFLite conversion is quite a challenge in this competition…
\nThe code snippets are actual code snippets from a notebook with a succeeding submission.
", "votes": 2, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2363144": "Some tips to prevent failing submissions\n\n1) Always start your preprocessing layer with\n```\ndata = data0[None]\n```\n\n2) Enforce correct shapes\n```\n# Crop/Pad\nN_FRAMES = len(data)\nif N_FRAMES > N_TARGET_FRAMES:\n data = tf.slice(data, [0,0], [N_TARGET_FRAMES, N_COLS])\nelif N_FRAMES < N_TARGET_FRAMES:\n data = tf.pad(data, [[0,N_TARGET_FRAMES-N_FRAMES], [0,0]])\n```\n\n3) Fill NaN Values\n```\n# Fill NaN\ndata = tf.where(tf.math.is_nan(data), tf.constant(0.0, data.dtype), data)\n```\n\nFor debugging purposes, you could try if you submission succeeds when only these 3 preprocessing steps are applied.", "2362265": "I have a model that keeps failing after 10 to 20 minutes after submitting.\n\nMy assumption is that it crashes on some sort of quirk on the dataset.\n\nI have successfully tan the tflite engine on all available datasets.\n\ni have also successfully ran it against dummy tensors [0, N], [1,N], ... [99999, N].\n\nDoes anyone have any insights of what else ai could try doing?\nThank you all 🙏" } }