{ "id": 433175, "title": "Inference time is not consistent", "url": "/competitions/asl-fingerspelling/discussion/433175", "author_name": "Wonjun", "post_date": "2023-08-20T15:55:26.725000", "votes": 1, "comment_count": 13, "views": 0, "content": "

In the exact same setting, sometimes inference time is fast and sometimes it is slow.
\nThere's up to a 2 times difference. Is anyone else experiencing the same issue?

", "messages": [ { "id": 2406236, "postDate": "2023-08-24T09:41:40.140Z", "content": "

Yes I also faced this problem as I have one same parameter model finish mostly in 5 hours ok, but it failed once due to overtime…

", "rawMarkdown": "Yes I also faced this problem as I have one same parameter model finish mostly in 5 hours ok, but it failed once due to overtime...", "votes": 1, "replies": [ { "id": 2406263, "postDate": "2023-08-24T10:14:33.470Z", "content": "

Does that mean that you must be more careful and allow for some margins in case it is \"crowded\"? Or a model that has succeeded will succeed for the final scoring?

", "rawMarkdown": "Does that mean that you must be more careful and allow for some margins in case it is \"crowded\"? Or a model that has succeeded will succeed for the final scoring?", "votes": 1, "replies": [ { "id": 2406271, "postDate": "2023-08-24T10:21:29.570Z", "content": "

I had a model which would sometimes finish under 5 hours submit and sometimes not. Therefore I changed it to finish in 4 hours (by making architecture smaller) and now it finishes submit everytime.

", "rawMarkdown": "I had a model which would sometimes finish under 5 hours submit and sometimes not. Therefore I changed it to finish in 4 hours (by making architecture smaller) and now it finishes submit everytime.", "votes": 1 }, { "id": 2406292, "postDate": "2023-08-24T10:38:47.983Z", "content": "

yes, mine be similar, but I have no time to retrain, so just try my luck. I only worried if server is crowded, will my job be affected to run longger.

", "rawMarkdown": "yes, mine be similar, but I have no time to retrain, so just try my luck. I only worried if server is crowded, will my job be affected to run longger.", "votes": 1 }, { "id": 2406349, "postDate": "2023-08-24T11:19:22.750Z", "content": "

Makes sense, thanks for the details!

", "rawMarkdown": "Makes sense, thanks for the details!" }, { "id": 2406474, "postDate": "2023-08-24T13:09:55.647Z", "content": "

@cdeotte Hi Chris wanna to confirm, what error did you see if overtime? Is it showing submission score error?

", "rawMarkdown": "@cdeotte Hi Chris wanna to confirm, what error did you see if overtime? Is it showing submission score error?", "votes": 1 }, { "id": 2406554, "postDate": "2023-08-24T13:59:11.133Z", "content": "

@goldenlock The overtime error does not say overtime. The overtime error says:

\n
\n

Submission Scoring Error
\n Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips

\n
\n

This is because at 5 hours and 20 minutes, Kaggle stops the running of our code and therefore the submission CSV file is not formed correctly. And we get this misleading error.

", "rawMarkdown": "@goldenlock The overtime error does not say overtime. The overtime error says:\n\n>Submission Scoring Error\nYour notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips\n\nThis is because at 5 hours and 20 minutes, Kaggle stops the running of our code and therefore the submission CSV file is not formed correctly. And we get this misleading error." }, { "id": 2406583, "postDate": "2023-08-24T14:24:07.453Z", "content": "

@cdeotte thanks! Your explanation helped a lot!

", "rawMarkdown": "@cdeotte thanks! Your explanation helped a lot!", "votes": 1 } ] } ] }, { "id": 2400016, "postDate": "2023-08-20T18:18:34.250Z", "content": "

I did observe it when inferencing on Kaggle's notebook. But it is very rare.

", "rawMarkdown": "I did observe it when inferencing on Kaggle's notebook. But it is very rare.", "votes": 1, "replies": [ { "id": 2400440, "postDate": "2023-08-21T04:35:48.303Z", "content": "

Thanks for your response.
\nFor me, it happens quite often.
\nBut I don't know why. I hope it doesn't affect submission inference time…

", "rawMarkdown": "Thanks for your response.\nFor me, it happens quite often.\nBut I don't know why. I hope it doesn't affect submission inference time...\n", "votes": 1 } ] }, { "id": 2399831, "postDate": "2023-08-20T15:55:26.727Z", "content": "

In the exact same setting, sometimes inference time is fast and sometimes it is slow.
\nThere's up to a 2 times difference. Is anyone else experiencing the same issue?

", "rawMarkdown": "In the exact same setting, sometimes inference time is fast and sometimes it is slow. \nThere's up to a 2 times difference. Is anyone else experiencing the same issue?", "votes": 1 }, { "id": 2401998, "postDate": "2023-08-22T00:27:05.580Z", "content": "

I have observed variation in model speed too. The following code with my model produced about 140ms when i committed just now. And about 350ms when i committed 2 hours ago with the same model (that's 2x difference!). I have also observed variation in submit time too. But i think variation in submit is less than variation in interactive and commit notebooks.

\n
%%timeit -n \n  prediction_fn(inputsframe)\n
", "rawMarkdown": "I have observed variation in model speed too. The following code with my model produced about 140ms when i **committed** just now. And about 350ms when i **committed** 2 hours ago **with the same model** (that's 2x difference!). I have also observed variation in **submit** time too. But i think variation in **submit** is less than variation in **interactive** and **commit** notebooks.\n\n %%timeit -n 10\n output = prediction_fn(inputs=frame)", "replies": [ { "id": 2404370, "postDate": "2023-08-23T07:55:21.253Z", "content": "

I also observed variation in submit is quite consistent.

", "rawMarkdown": "I also observed variation in submit is quite consistent." } ] }, { "id": 2401827, "postDate": "2023-08-21T20:16:36.173Z", "content": "

It might be due to Kaggle's infrastructure? Maybe it is temporary? When did you experience this?

", "rawMarkdown": "It might be due to Kaggle's infrastructure? Maybe it is temporary? When did you experience this?" } ], "comments": [ { "id": 2406236, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T09:41:40.140000", "content": "

Yes I also faced this problem as I have one same parameter model finish mostly in 5 hours ok, but it failed once due to overtime…

", "votes": 1, "replies": [ { "id": 2406263, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2023-08-24T10:14:33.470000", "content": "

Does that mean that you must be more careful and allow for some margins in case it is \"crowded\"? Or a model that has succeeded will succeed for the final scoring?

", "votes": 1, "replies": [ { "id": 2406271, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-08-24T10:21:29.570000", "content": "

I had a model which would sometimes finish under 5 hours submit and sometimes not. Therefore I changed it to finish in 4 hours (by making architecture smaller) and now it finishes submit everytime.

", "votes": 1, "replies": [] }, { "id": 2406292, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T10:38:47.983000", "content": "

yes, mine be similar, but I have no time to retrain, so just try my luck. I only worried if server is crowded, will my job be affected to run longger.

", "votes": 1, "replies": [] }, { "id": 2406349, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2023-08-24T11:19:22.750000", "content": "

Makes sense, thanks for the details!

", "votes": 0, "replies": [] }, { "id": 2406474, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T13:09:55.647000", "content": "

@cdeotte Hi Chris wanna to confirm, what error did you see if overtime? Is it showing submission score error?

", "votes": 1, "replies": [] }, { "id": 2406554, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-08-24T13:59:11.133000", "content": "

@goldenlock The overtime error does not say overtime. The overtime error says:

\n
\n

Submission Scoring Error
\n Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips

\n
\n

This is because at 5 hours and 20 minutes, Kaggle stops the running of our code and therefore the submission CSV file is not formed correctly. And we get this misleading error.

", "votes": 0, "replies": [] }, { "id": 2406583, "author_name": "gezi", "author_url": "", "post_date": "2023-08-24T14:24:07.453000", "content": "

@cdeotte thanks! Your explanation helped a lot!

", "votes": 1, "replies": [] } ] } ] }, { "id": 2400016, "author_name": "Yu Wu", "author_url": "", "post_date": "2023-08-20T18:18:34.250000", "content": "

I did observe it when inferencing on Kaggle's notebook. But it is very rare.

", "votes": 1, "replies": [ { "id": 2400440, "author_name": "Wonjun", "author_url": "", "post_date": "2023-08-21T04:35:48.303000", "content": "

Thanks for your response.
\nFor me, it happens quite often.
\nBut I don't know why. I hope it doesn't affect submission inference time…

", "votes": 1, "replies": [] } ] }, { "id": 2401998, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-08-22T00:27:05.580000", "content": "

I have observed variation in model speed too. The following code with my model produced about 140ms when i committed just now. And about 350ms when i committed 2 hours ago with the same model (that's 2x difference!). I have also observed variation in submit time too. But i think variation in submit is less than variation in interactive and commit notebooks.

\n
%%timeit -n \n  prediction_fn(inputsframe)\n
", "votes": 0, "replies": [ { "id": 2404370, "author_name": "Wonjun", "author_url": "", "post_date": "2023-08-23T07:55:21.253000", "content": "

I also observed variation in submit is quite consistent.

", "votes": 0, "replies": [] } ] }, { "id": 2401827, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2023-08-21T20:16:36.173000", "content": "

It might be due to Kaggle's infrastructure? Maybe it is temporary? When did you experience this?

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2406236": "Yes I also faced this problem as I have one same parameter model finish mostly in 5 hours ok, but it failed once due to overtime...", "2400016": "I did observe it when inferencing on Kaggle's notebook. But it is very rare.", "2399831": "In the exact same setting, sometimes inference time is fast and sometimes it is slow. \nThere's up to a 2 times difference. Is anyone else experiencing the same issue?", "2401998": "I have observed variation in model speed too. The following code with my model produced about 140ms when i **committed** just now. And about 350ms when i **committed** 2 hours ago **with the same model** (that's 2x difference!). I have also observed variation in **submit** time too. But i think variation in **submit** is less than variation in **interactive** and **commit** notebooks.\n\n %%timeit -n 10\n output = prediction_fn(inputs=frame)", "2401827": "It might be due to Kaggle's infrastructure? Maybe it is temporary? When did you experience this?" } }