Dear organisers from Google and/or Kaggle, @thadstarner @sohier
\nWe are now 11 days into this competition, and it is still not possible to get any submissions through the scoring system.
\nIt is also not possible to verify one's own submissions in the current Kaggle kernel since tflite-runtime can not be installed due to version conflicts. In order to fix this, I have posted a notebook with a pinned kernel from the previous Kaggle competition here.
\nMany of us have meanwhile created a dummy \"random\" submission model that can successfully do inference in a kernel that does support tflite-runtime (either a local install or a pinned kernel from the previous competition) and based on this we are quite confident that we have established the right input and output formats (using experience from the previous competition, we assumed that REQUIRED_SIGNATURE and REQUIRED_OUTPUT had the same meaning as before, but even that is not confirmed).
\nHowever, consistently, all submissions keep failing, and we get no information regarding the reason(s) for this. Most submissions fail after 1 or 2 minutes. Some submissions seem to be scoring for a long time (~2 hour), but the current understanding is that they somehow get stuck in a queue, since resubmission of exactly the same model does not result in consistent behaviour w.r.t. \"time before failure\".
\nOverall, we are getting quite frustrated and really want to insist on the organisers to clear up the issues and clarify the remaining misunderstandings, so we can finally start this competition.
\nWe sincerely hope that the organisers from Google and/or Kaggle will take the time to test-run the scoring pipeline using a dummy submission of their own.
The most convenient solution would be to share the resulting script that **has been verified to create a dummy submission that successfully passes scoring. ** If that is not possible, then at least ensure that successful submissions CAN be made and confirm that our assumptions regarding the requirements made to the model are correct, i.e.:
\n**Input format: **
\ntf.keras.Input(shape=(NUM_FEATURES), dtype=tf.float32, name=\"inputs\")
\n(batch dimension is used for #frames, required name is \"inputs\")
**Output shape: ** [None,59]
\n(batch dimension is used for #frames, required name is \"outputs\")
**Requirements on length of output sequence: ** ???
\n(same as input length? same or smaller than input length? no restrictions?
Dear organisers from Google and/or Kaggle, @thadstarner @sohier
\nWe are now 11 days into this competition, and it is still not possible to get any submissions through the scoring system.
\nIt is also not possible to verify one's own submissions in the current Kaggle kernel since tflite-runtime can not be installed due to version conflicts. In order to fix this, I have posted a notebook with a pinned kernel from the previous Kaggle competition here.
\nMany of us have meanwhile created a dummy \"random\" submission model that can successfully do inference in a kernel that does support tflite-runtime (either a local install or a pinned kernel from the previous competition) and based on this we are quite confident that we have established the right input and output formats (using experience from the previous competition, we assumed that REQUIRED_SIGNATURE and REQUIRED_OUTPUT had the same meaning as before, but even that is not confirmed).
\nHowever, consistently, all submissions keep failing, and we get no information regarding the reason(s) for this. Most submissions fail after 1 or 2 minutes. Some submissions seem to be scoring for a long time (~2 hour), but the current understanding is that they somehow get stuck in a queue, since resubmission of exactly the same model does not result in consistent behaviour w.r.t. \"time before failure\".
\nOverall, we are getting quite frustrated and really want to insist on the organisers to clear up the issues and clarify the remaining misunderstandings, so we can finally start this competition.
\nWe sincerely hope that the organisers from Google and/or Kaggle will take the time to test-run the scoring pipeline using a dummy submission of their own.
The most convenient solution would be to share the resulting script that **has been verified to create a dummy submission that successfully passes scoring. ** If that is not possible, then at least ensure that successful submissions CAN be made and confirm that our assumptions regarding the requirements made to the model are correct, i.e.:
\n**Input format: **
\ntf.keras.Input(shape=(NUM_FEATURES), dtype=tf.float32, name=\"inputs\")
\n(batch dimension is used for #frames, required name is \"inputs\")
**Output shape: ** [None,59]
\n(batch dimension is used for #frames, required name is \"outputs\")
**Requirements on length of output sequence: ** ???
\n(same as input length? same or smaller than input length? no restrictions?
The competition should be reopened when there is no submission bugs..🤕
", "rawMarkdown": "The competition should be reopened when there is no submission bugs..🤕", "votes": 3 }, { "id": 2270817, "postDate": "2023-05-23T12:54:59.107Z", "content": "I suggest that Kaggle provides more detailed debug information specifically for this competition during submission, or allows us to create a blank sample_submission.csv file. This could help us break through the submission limits. Spending too much time on submissions is really frustrating.
", "rawMarkdown": "I suggest that Kaggle provides more detailed debug information specifically for this competition during submission, or allows us to create a blank sample_submission.csv file. This could help us break through the submission limits. Spending too much time on submissions is really frustrating.", "votes": 2, "replies": [ { "id": 2271052, "postDate": "2023-05-23T15:35:01.160Z", "content": "\nIn this competition, we need to submit a model, not a .csv file. This model is then applied on the test data by the organisers and the results are scored.
\nSo as long as the organisers do not remove the problems in the scoring pipeline and/or clarify which information we are missing (i.e., why all submissions are failing), this competition is officially \"dead\". Except maybe for a few new participants, most of us have stopped trying to submit entirely, while some are probably still developing models based on a train-validate split.
\nI, for one, am very disappointed in the lack of response of the organisers, and find this very shameful
", "rawMarkdown": "@dwchen \n\nIn this competition, we need to submit a model, not a .csv file. This model is then applied on the test data by the organisers and the results are scored.\n\nSo as long as the organisers do not remove the problems in the scoring pipeline and/or clarify which information we are missing (i.e., why all submissions are failing), **this competition is officially \"dead\"**. Except maybe for a few new participants, most of us have stopped trying to submit entirely, while some are probably still developing models based on a train-validate split.\n\nI, for one, am very disappointed in the lack of response of the organisers, and find this very shameful", "votes": 5 } ] } ], "comments": [ { "id": 2270066, "author_name": "arutema47", "author_url": "", "post_date": "2023-05-23T01:02:15.760000", "content": "The competition should be reopened when there is no submission bugs..🤕
", "votes": 3, "replies": [] }, { "id": 2270817, "author_name": "Dewei Chen", "author_url": "", "post_date": "2023-05-23T12:54:59.107000", "content": "I suggest that Kaggle provides more detailed debug information specifically for this competition during submission, or allows us to create a blank sample_submission.csv file. This could help us break through the submission limits. Spending too much time on submissions is really frustrating.
", "votes": 2, "replies": [ { "id": 2271052, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-05-23T15:35:01.160000", "content": "\nIn this competition, we need to submit a model, not a .csv file. This model is then applied on the test data by the organisers and the results are scored.
\nSo as long as the organisers do not remove the problems in the scoring pipeline and/or clarify which information we are missing (i.e., why all submissions are failing), this competition is officially \"dead\". Except maybe for a few new participants, most of us have stopped trying to submit entirely, while some are probably still developing models based on a train-validate split.
\nI, for one, am very disappointed in the lack of response of the organisers, and find this very shameful
", "votes": 5, "replies": [] } ] } ], "raw_markdown_by_id": { "2267991": "Dear organisers from Google and/or Kaggle, @thadstarner @sohier \n\nWe are now 11 days into this competition, and it is still not possible to get any submissions through the scoring system.\n\nIt is also not possible to verify one's own submissions in the current Kaggle kernel since tflite-runtime can not be installed due to version conflicts. In order to fix this, I have posted a notebook with a pinned kernel from the previous Kaggle competition [here](https://www.kaggle.com/code/wonderingalice/dummy-submission-with-previous-kernel?kernelSessionId=130400757).\n\nMany of us have meanwhile created a dummy \"random\" submission model that can successfully do inference in a kernel that does support tflite-runtime (either a local install or a pinned kernel from the previous competition) and based on this we are quite confident that we have established the right input and output formats (using experience from the previous competition, we assumed that REQUIRED_SIGNATURE and REQUIRED_OUTPUT had the same meaning as before, but even that is not confirmed). \n\n**However, consistently, all submissions keep failing, and we get no information regarding the reason(s) for this.** Most submissions fail after 1 or 2 minutes. Some submissions seem to be scoring for a long time (~2 hour), but the current understanding is that they somehow get stuck in a queue, since resubmission of exactly the same model does not result in consistent behaviour w.r.t. \"time before failure\".\n\nOverall, we are getting quite frustrated and really want to insist on the organisers to clear up the issues and clarify the remaining misunderstandings, so we can finally start this competition. \nWe sincerely hope that the organisers from Google and/or Kaggle will take the time to test-run the scoring pipeline using a dummy submission of their own.\n\nThe most convenient solution would be to share the resulting script that **has been verified to create a dummy submission that successfully passes scoring. ** If that is not possible, then at least ensure that successful submissions CAN be made and confirm that our assumptions regarding the requirements made to the model are correct, i.e.:\n\n**Input format: **\ntf.keras.Input(shape=(NUM_FEATURES), dtype=tf.float32, name=\"inputs\")\n(batch dimension is used for #frames, required name is \"inputs\")\n\n**Output shape: ** [None,59]\n(batch dimension is used for #frames, required name is \"outputs\")\n\n**Requirements on length of output sequence: ** ???\n(same as input length? same or smaller than input length? no restrictions?\n\n\n", "2270066": "The competition should be reopened when there is no submission bugs..🤕", "2270817": "I suggest that Kaggle provides more detailed debug information specifically for this competition during submission, or allows us to create a blank sample_submission.csv file. This could help us break through the submission limits. Spending too much time on submissions is really frustrating." } }