I have two submits with same arch ctc models which used to pass (have not failed before,used to finish in about 4h55min),
\n but today they all fail after running about 5h 20min.
\nShowing 'Submission Scoring Error', is it due to overtime?
\nOr some other mistakes inside the tflite model?
Update the status, this is not kaggle platform problem, I'm not sure what did I changed but just tested the model pass is 8.2s/it while not pass is 9s/it
", "rawMarkdown": "Update the status, this is not kaggle platform problem, I'm not sure what did I changed but just tested the model pass is 8.2s/it while not pass is 9s/it", "votes": 1 }, { "id": 2406505, "postDate": "2023-08-24T13:28:10.327Z", "content": "I have two submits with same arch ctc models which used to pass (have not failed before,used to finish in about 4h55min),
\n but today they all fail after running about 5h 20min.
\nShowing 'Submission Scoring Error', is it due to overtime?
\nOr some other mistakes inside the tflite model?
Yes. I've submitted large models to test the maximal acceptable inference time. When average inference time > 0.8s/sample, I got \"Submission Scoring Error\"
", "rawMarkdown": "Yes. I've submitted large models to test the maximal acceptable inference time. When average inference time > 0.8s/sample, I got \"Submission Scoring Error\"", "replies": [ { "id": 2406546, "postDate": "2023-08-24T13:49:35.167Z", "content": "Thanks @nightsh4de .I have 1 model with 0.81s/sample which used to pass, but fail today😅
\nI have no time to retrain smaller models.., so just by luck to submit the last two models.. so sad.
In previous competition all successful submits were valid. I think here is the same logic and you shouldn't worry about successful ones.
\nAnd due to some hardware (I guess) issues same submissions may have different scoring time. Best way just to submit it again (maybe at night or early morning. but it's already not the case at the last day)
", "rawMarkdown": "In previous competition all successful submits were valid. I think here is the same logic and you shouldn't worry about successful ones.\n\nAnd due to some hardware (I guess) issues same submissions may have different scoring time. Best way just to submit it again (maybe at night or early morning. but it's already not the case at the last day)", "replies": [ { "id": 2406529, "postDate": "2023-08-24T13:38:23.177Z", "content": "Thanks @kolyaforrat If it is as you said, I wasted two valuable submits😂, I thought the server might running slower then before.. but not sure, why not show notebook overtime but show 'scoring error'? Confusing
", "rawMarkdown": "Thanks @kolyaforrat If it is as you said, I wasted two valuable submits😂, I thought the server might running slower then before.. but not sure, why not show notebook overtime but show 'scoring error'? Confusing", "replies": [ { "id": 2406538, "postDate": "2023-08-24T13:43:42.653Z", "content": "As I remember we talked about this some time ago :)
\nThis type of competition different with others. Usually we have 9 hours to kernel runtime (with anything, inference, train, etc). But here it indifferent to your kernel runtime, you can only copy your .tflite file or convert model or train it, the time calculation starts after the kernel finished. In typical Kaggle competition it's a stage where Kaggle just compute metrics, here they also use their hidden inference pipeline to generate predictions. That's why Kaggle just don't have some specific error message for this situation.
", "rawMarkdown": "As I remember we talked about this some time ago :)\n\nThis type of competition different with others. Usually we have 9 hours to kernel runtime (with anything, inference, train, etc). But here it indifferent to your kernel runtime, you can only copy your .tflite file or convert model or train it, the time calculation starts after the kernel finished. In typical Kaggle competition it's a stage where Kaggle just compute metrics, here they also use their hidden inference pipeline to generate predictions. That's why Kaggle just don't have some specific error message for this situation.\n", "votes": 1 }, { "id": 2406541, "postDate": "2023-08-24T13:46:42.947Z", "content": "Very clear, thanks so much @kolyaforrat .
", "rawMarkdown": "Very clear, thanks so much @kolyaforrat .", "votes": 1 }, { "id": 2406837, "postDate": "2023-08-24T17:15:44.393Z", "content": "for me when the submission fails it would waste me 8h GPU time and I have way too many submission not limited GPU. Do you have any tip on how to reuse, or quickly resubmit the model without wasting another 8h GPU? Thanks!
", "rawMarkdown": "for me when the submission fails it would waste me 8h GPU time and I have way too many submission not limited GPU. Do you have any tip on how to reuse, or quickly resubmit the model without wasting another 8h GPU? Thanks!" }, { "id": 2406889, "postDate": "2023-08-24T17:37:33.083Z", "content": "@yubinxie for submission you do not need to turn on gpu.
", "rawMarkdown": "@yubinxie for submission you do not need to turn on gpu." } ] } ] }, { "id": 2406712, "postDate": "2023-08-24T15:41:21.170Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2406746, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T16:07:17.030000", "content": "Update the status, this is not kaggle platform problem, I'm not sure what did I changed but just tested the model pass is 8.2s/it while not pass is 9s/it
", "votes": 1, "replies": [] }, { "id": 2406544, "author_name": "Yu Wu", "author_url": "", "post_date": "2023-08-24T13:48:31.403000", "content": "Yes. I've submitted large models to test the maximal acceptable inference time. When average inference time > 0.8s/sample, I got \"Submission Scoring Error\"
", "votes": 0, "replies": [ { "id": 2406546, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T13:49:35.167000", "content": "Thanks @nightsh4de .I have 1 model with 0.81s/sample which used to pass, but fail today😅
\nI have no time to retrain smaller models.., so just by luck to submit the last two models.. so sad.
In previous competition all successful submits were valid. I think here is the same logic and you shouldn't worry about successful ones.
\nAnd due to some hardware (I guess) issues same submissions may have different scoring time. Best way just to submit it again (maybe at night or early morning. but it's already not the case at the last day)
", "votes": 0, "replies": [ { "id": 2406529, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T13:38:23.177000", "content": "Thanks @kolyaforrat If it is as you said, I wasted two valuable submits😂, I thought the server might running slower then before.. but not sure, why not show notebook overtime but show 'scoring error'? Confusing
", "votes": 0, "replies": [ { "id": 2406538, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-08-24T13:43:42.653000", "content": "As I remember we talked about this some time ago :)
\nThis type of competition different with others. Usually we have 9 hours to kernel runtime (with anything, inference, train, etc). But here it indifferent to your kernel runtime, you can only copy your .tflite file or convert model or train it, the time calculation starts after the kernel finished. In typical Kaggle competition it's a stage where Kaggle just compute metrics, here they also use their hidden inference pipeline to generate predictions. That's why Kaggle just don't have some specific error message for this situation.
", "votes": 1, "replies": [] }, { "id": 2406541, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T13:46:42.947000", "content": "Very clear, thanks so much @kolyaforrat .
", "votes": 1, "replies": [] }, { "id": 2406837, "author_name": "Yoobin", "author_url": "", "post_date": "2023-08-24T17:15:44.393000", "content": "for me when the submission fails it would waste me 8h GPU time and I have way too many submission not limited GPU. Do you have any tip on how to reuse, or quickly resubmit the model without wasting another 8h GPU? Thanks!
", "votes": 0, "replies": [] }, { "id": 2406889, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T17:37:33.083000", "content": "@yubinxie for submission you do not need to turn on gpu.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2406712, "author_name": "", "author_url": "", "post_date": "2023-08-24T15:41:21.170000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2406746": "Update the status, this is not kaggle platform problem, I'm not sure what did I changed but just tested the model pass is 8.2s/it while not pass is 9s/it", "2406505": "I have two submits with same arch ctc models which used to pass (have not failed before,used to finish in about 4h55min),\n but today they all fail after running about 5h 20min. \nShowing 'Submission Scoring Error', is it due to overtime? \nOr some other mistakes inside the tflite model?", "2406544": "Yes. I've submitted large models to test the maximal acceptable inference time. When average inference time > 0.8s/sample, I got \"Submission Scoring Error\"", "2406514": "In previous competition all successful submits were valid. I think here is the same logic and you shouldn't worry about successful ones.\n\nAnd due to some hardware (I guess) issues same submissions may have different scoring time. Best way just to submit it again (maybe at night or early morning. but it's already not the case at the last day)", "2406712": "" } }