Dear competition organisers,
\n@sohier
We are now 3 weeks into this competition and only 8 people have managed to make meaningful submissions (i.e. submissions that are not dummy or baseline).
\nThe main reason for this is that most people's contributions keep failing the scoring pipeline for reasons that can not be explained by the instructions provided by the organisers.
\nMeanwhile we have examples that allow the successful submission of simple (dummy) models and baselines, but attempts to implement the required pre- and postprocessing do not pass scoring. In particular, solutions that run perfectly well in the prescribed TfLite runtime v2.9.1 in our notebooks fail scoring, only resulting in an uninformative \"Scoring error\". This also applies to preprocessing solutions that worked perfectly in the previous (GISLR) competition.
\nThis situation basically reduces all attempts to random guessing (at a rate of 5 tries a day). Since we cannot debug our Tflite models locally or in Kaggle kernels (using only TfLite runtime v2.9.1) my request to the organisers is to at least provide us with information about the additional restrictions that are obviously in place but have not been communicated!
", "messages": [ { "id": 2282263, "postDate": "2023-05-31T13:37:19.173Z", "content": "Dear competition organisers,
\n@sohier
We are now 3 weeks into this competition and only 8 people have managed to make meaningful submissions (i.e. submissions that are not dummy or baseline).
\nThe main reason for this is that most people's contributions keep failing the scoring pipeline for reasons that can not be explained by the instructions provided by the organisers.
\nMeanwhile we have examples that allow the successful submission of simple (dummy) models and baselines, but attempts to implement the required pre- and postprocessing do not pass scoring. In particular, solutions that run perfectly well in the prescribed TfLite runtime v2.9.1 in our notebooks fail scoring, only resulting in an uninformative \"Scoring error\". This also applies to preprocessing solutions that worked perfectly in the previous (GISLR) competition.
\nThis situation basically reduces all attempts to random guessing (at a rate of 5 tries a day). Since we cannot debug our Tflite models locally or in Kaggle kernels (using only TfLite runtime v2.9.1) my request to the organisers is to at least provide us with information about the additional restrictions that are obviously in place but have not been communicated!
", "rawMarkdown": "Dear competition organisers,\n@sohier \n\nWe are now 3 weeks into this competition and **only 8 people** have managed to make meaningful submissions (i.e. submissions that are not dummy or baseline).\n\nThe main reason for this is that **most people's contributions keep failing the scoring pipeline** for reasons that **can not be explained by the instructions provided by the organisers**.\n\nMeanwhile we have examples that allow the successful submission of simple (dummy) models and baselines, but attempts to implement the required pre- and postprocessing do not pass scoring. In particular, solutions that run perfectly well in the prescribed TfLite runtime v2.9.1 in our notebooks fail scoring, only resulting in an uninformative \"Scoring error\". This also applies to preprocessing solutions that worked perfectly in the previous (GISLR) competition. \n\nThis situation basically reduces all attempts to random guessing (at a rate of 5 tries a day). Since we cannot debug our Tflite models locally or in Kaggle kernels (using only TfLite runtime v2.9.1) my request to the organisers is to at least **provide us with information about the additional restrictions that are obviously in place but have not been communicated!**\n", "votes": 14 }, { "id": 2282553, "postDate": "2023-05-31T16:31:05.697Z", "content": "I agree with you, participants need some clarification if there some additional restrictions
\nBut in Code page we have this kernel by @irohith with not so much views and upvotes.
\nThis is not a dummy model and for example my TFLite model code is almost the same. I think everyone who still got errors should look through it, maybe it will be helpful
BTW, I just tried and even with the oldest version that supports Python 3.7, i.e. TfLite runtime 2.7.0 ( and using tflite model generated by Tensorflow 2.11.0) I get no errors when running inference while submission scoring fails!
", "rawMarkdown": "BTW, I just tried and even with the oldest version that supports Python 3.7, i.e. **TfLite runtime 2.7.0** ( and using tflite model generated by Tensorflow 2.11.0) I get no errors when running inference while submission scoring fails!", "votes": 1 }, { "id": 2282574, "postDate": "2023-05-31T16:49:48.123Z", "content": "I'll look into this.
", "rawMarkdown": "I'll look into this.", "votes": 2, "replies": [ { "id": 2282861, "postDate": "2023-05-31T21:36:17.587Z", "content": "I've posted an update in the related thread here: https://www.kaggle.com/competitions/asl-fingerspelling/discussion/413598#2282859
", "rawMarkdown": "I've posted an update in the related thread here: https://www.kaggle.com/competitions/asl-fingerspelling/discussion/413598#2282859", "replies": [ { "id": 2284773, "postDate": "2023-06-02T09:11:24.987Z", "content": "Unfortunately I don't think this update addresses concerns at all. The evaluation procedure doesn't seem to match the one listed on the Evaluation page - this is the crux of the issue.
", "rawMarkdown": "Unfortunately I don't think this update addresses concerns at all. The evaluation procedure doesn't seem to match the one listed on the Evaluation page - this is the crux of the issue.", "votes": 1 }, { "id": 2284797, "postDate": "2023-06-02T09:25:02.700Z", "content": "In fact the update only clarifies things we had already figured out ourselves.
", "rawMarkdown": "In fact the update only clarifies things we had already figured out ourselves." } ] } ] }, { "id": 2283178, "postDate": "2023-06-01T05:31:35.983Z", "content": "Think a time out on all the requests for provided solutions is needed. Know I have responded with some answers as have others.
\nMost competitions have some one or two notebooks that dominate and others just fork. It discourages original solutions.
\nIt is refreshing to have a competition to be able to work on and resolve issues, take advantage of that. What you learn on your own is more valuable. Kaggle should be a place to learn. It has been clear for awhile now that actual submissions were possible.
\nPeople may be working on other things, and this has 2 months to go. Take a break, do some research on your own.
\n\nMost competitions have some one or two notebooks that dominate and others just fork. It discourages original solutions.
\n
On the contrary, I feel that most models not working and being unable to explain why forces people to the few \"known good\" architectures in public Kernels. I personally have a much better model locally but I can't submit it. The main issue here is that the competition specification doesn't match the actual competition.
", "rawMarkdown": "> Most competitions have some one or two notebooks that dominate and others just fork. It discourages original solutions.\n\nOn the contrary, I feel that most models not working and being unable to explain why forces people to the few \"known good\" architectures in public Kernels. I personally have a much better model locally but I can't submit it. The main issue here is that the competition specification doesn't match the actual competition.", "votes": 3, "replies": [ { "id": 2284796, "postDate": "2023-06-02T09:23:46.417Z", "content": "Hi @anokas
\nwe all do, and the organisers @sohier ignore or refuse to answer to this issue!
\nI'm beginning to wonder whether the real purpose of this competition might be to collect data on how the community approaches a black box debugging task …
I agree that public high scoring notebooks can make competitions less interesting. I also agree that these competitions are interesting learning experiences. I've learned a lot from the previous ASL competition, on model tuning but also on porting from PyTorch to Keras and TFLite.
\nHowever, running a notebook locally and on Kaggle with settings that should work (according to the evaluation description page) just to have it fail during submission without knowing why is not a learning experience. If there is no feedback at all on why the submission failed, there is nothing to learn.
\nThere have been comments that adding the most basic operations like tf.reduce_sum of tf.expand_dims, which work locally and in the Kaggle notebooks, can make submissions fail. This should not be the case if everything is working correctly.
I, like @anokas, also have a model that performs quite well, but I can't submit it. I've been taking a methodological approach to debugging (if you can call it that) the submission pipeline, using up my 5 submissions a day, but this is frustrating because it is not what this competition should be about. As a researcher in the field, I want to see interesting solutions come out of this competition, and these submission problems are reducing the amount of actual contributions instead of people forking the one or two notebooks that do work.
", "rawMarkdown": "I agree that public high scoring notebooks can make competitions less interesting. I also agree that these competitions are interesting learning experiences. I've learned a lot from the previous ASL competition, on model tuning but also on porting from PyTorch to Keras and TFLite.\n\nHowever, running a notebook locally and on Kaggle with settings that should work (according to the evaluation description page) just to have it fail during submission without knowing why is not a learning experience. If there is no feedback at all on why the submission failed, there is nothing to learn.\n\nThere have been comments that adding the most basic operations like `tf.reduce_sum` of `tf.expand_dims`, which work locally and in the Kaggle notebooks, can make submissions fail. This should not be the case if everything is working correctly.\n\nI, like @anokas, also have a model that performs quite well, but I can't submit it. I've been taking a methodological approach to debugging (if you can call it that) the submission pipeline, using up my 5 submissions a day, but this is frustrating because it is not what this competition should be about. As a researcher in the field, I want to see interesting solutions come out of this competition, and these submission problems are reducing the amount of actual contributions instead of people forking the one or two notebooks that *do* work.", "votes": 3, "replies": [ { "id": 2286012, "postDate": "2023-06-03T07:07:57.593Z", "content": "Figured there would be pushback on this and the petty downvotes. But the point being made was in reference to this post. and this. George Megre and Forrato have been able to make successful submissions over 9 days ago and put explanations in their comments. Both are using pipelines similar to the previous competition and there are code snippets here from them as well as a provided notebook link above. It is possible to consider their example and to look at the discussions to help find solutions. e.g. from Forrato:
\n\"About Conv2D and TFLite you can read in comments to my team's write-up from previous competition.
\nTLDR: not all conv types are good convertable to TFLite with torch-onnx-keras pipeline. But all are convertable with nobuco. But in Kaggle environment some of them slower than our keras implemetation (But locally in different environment nobuco is better). So with nobuco no errors but slower\"
In reference to learning, it is also possible to try working with Tensorflow rather than Pytorch if there are issues with Pytorch or onnx.
\nThe latest post that the competition evaluation is using TFlite runtime 2.14 seems bizarre since that does not appear to be available anywhere. So clarification on where that is to be obtained has been asked here
", "rawMarkdown": "Figured there would be pushback on this and the petty downvotes. But the point being made was in reference to [this post](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/409722). [and this](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/413598). George Megre and Forrato have been able to make successful submissions over 9 days ago and put explanations in their comments. Both are using pipelines similar to the previous competition and there are code snippets here from them as well as a provided notebook link above. It is possible to consider their example and to look at the discussions to help find solutions. e.g. from Forrato:\n\"About Conv2D and TFLite you can read in comments to my team's write-up from previous competition.\nTLDR: not all conv types are good convertable to TFLite with torch-onnx-keras pipeline. But all are convertable with nobuco. But in Kaggle environment some of them slower than our keras implemetation (But locally in different environment nobuco is better). So with nobuco no errors but slower\"\n\nIn reference to learning, it is also possible to try working with Tensorflow rather than Pytorch if there are issues with Pytorch or onnx. \n\nThe latest post that the competition evaluation is using TFlite runtime 2.14 seems bizarre since that does not appear to be available anywhere. So clarification on where that is to be obtained has been asked [here](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/414682)\n" }, { "id": 2286134, "postDate": "2023-06-03T08:45:00.090Z", "content": "Hey @something4kag
\nI am also using the exact same pipeline from the previous competition which worked there and where we also had to take into account the Conv conversion issues, so the code should not be the issue, the issue is with the environment.
\nI am also, like many others, unable to submit very simple models with Tensorflow code (so no PyTorch conversion).
\nSee for example this post from @markwijkhuizen https://www.kaggle.com/competitions/asl-fingerspelling/discussion/414631 who also got a good score on the previous competition.
\nI've also looked in detail at the posts with solutions you posted, and unfortunately those do not seem to solve my issues. We're not just complaining here, we're trying to find a solution that works for everyone :)
\nFor getting version 2.14, check my reply to your comment. Note that you will get an error if you import both tensorflow and tflite-runtime with version 2.14.
", "rawMarkdown": "Hey @something4kag \n\nI am also using the exact same pipeline from the previous competition which worked there and where we also had to take into account the Conv conversion issues, so the code should not be the issue, the issue is with the environment.\n\nI am also, like many others, unable to submit very simple models with Tensorflow code (so no PyTorch conversion).\n\nSee for example this post from @markwijkhuizen https://www.kaggle.com/competitions/asl-fingerspelling/discussion/414631 who also got a good score on the previous competition.\n\nI've also looked in detail at the posts with solutions you posted, and unfortunately those do not seem to solve my issues. We're not just complaining here, we're trying to find a solution that works for everyone :)\n\nFor getting version 2.14, check my reply to your comment. Note that you will get an error if you import both tensorflow and tflite-runtime with version 2.14.", "votes": 1 }, { "id": 2286205, "postDate": "2023-06-03T09:50:06.733Z", "content": "I have added a comment in the post on TFLite Runtime Version number correction
\nhttps://www.kaggle.com/competitions/asl-fingerspelling/discussion/414682
\nfor a way to test your submission.zip with 2.14.0
\nFor the moment it is best to test your model this way, because of the error
\nImportError: generic_type: type \"InterpreterWrapper\" is already registered!
\nif you try to import both Tensorflow and TFLite runtime in the same notebook.
\nIt is a prerelease and dev version so perhaps later will be fixed up. There are ways to do it, but think this is a simple way to get an idea if your model will work or not.
I agree with you, participants need some clarification if there some additional restrictions
\nBut in Code page we have this kernel by @irohith with not so much views and upvotes.
\nThis is not a dummy model and for example my TFLite model code is almost the same. I think everyone who still got errors should look through it, maybe it will be helpful
BTW, I just tried and even with the oldest version that supports Python 3.7, i.e. TfLite runtime 2.7.0 ( and using tflite model generated by Tensorflow 2.11.0) I get no errors when running inference while submission scoring fails!
", "votes": 1, "replies": [] }, { "id": 2282574, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-05-31T16:49:48.123000", "content": "I'll look into this.
", "votes": 2, "replies": [ { "id": 2282861, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-05-31T21:36:17.587000", "content": "I've posted an update in the related thread here: https://www.kaggle.com/competitions/asl-fingerspelling/discussion/413598#2282859
", "votes": 0, "replies": [ { "id": 2284773, "author_name": "anokas", "author_url": "", "post_date": "2023-06-02T09:11:24.987000", "content": "Unfortunately I don't think this update addresses concerns at all. The evaluation procedure doesn't seem to match the one listed on the Evaluation page - this is the crux of the issue.
", "votes": 1, "replies": [] }, { "id": 2284797, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-06-02T09:25:02.700000", "content": "In fact the update only clarifies things we had already figured out ourselves.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2283178, "author_name": "something4kag", "author_url": "", "post_date": "2023-06-01T05:31:35.983000", "content": "Think a time out on all the requests for provided solutions is needed. Know I have responded with some answers as have others.
\nMost competitions have some one or two notebooks that dominate and others just fork. It discourages original solutions.
\nIt is refreshing to have a competition to be able to work on and resolve issues, take advantage of that. What you learn on your own is more valuable. Kaggle should be a place to learn. It has been clear for awhile now that actual submissions were possible.
\nPeople may be working on other things, and this has 2 months to go. Take a break, do some research on your own.
\n\nMost competitions have some one or two notebooks that dominate and others just fork. It discourages original solutions.
\n
On the contrary, I feel that most models not working and being unable to explain why forces people to the few \"known good\" architectures in public Kernels. I personally have a much better model locally but I can't submit it. The main issue here is that the competition specification doesn't match the actual competition.
", "votes": 3, "replies": [ { "id": 2284796, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-06-02T09:23:46.417000", "content": "Hi @anokas
\nwe all do, and the organisers @sohier ignore or refuse to answer to this issue!
\nI'm beginning to wonder whether the real purpose of this competition might be to collect data on how the community approaches a black box debugging task …
I agree that public high scoring notebooks can make competitions less interesting. I also agree that these competitions are interesting learning experiences. I've learned a lot from the previous ASL competition, on model tuning but also on porting from PyTorch to Keras and TFLite.
\nHowever, running a notebook locally and on Kaggle with settings that should work (according to the evaluation description page) just to have it fail during submission without knowing why is not a learning experience. If there is no feedback at all on why the submission failed, there is nothing to learn.
\nThere have been comments that adding the most basic operations like tf.reduce_sum of tf.expand_dims, which work locally and in the Kaggle notebooks, can make submissions fail. This should not be the case if everything is working correctly.
I, like @anokas, also have a model that performs quite well, but I can't submit it. I've been taking a methodological approach to debugging (if you can call it that) the submission pipeline, using up my 5 submissions a day, but this is frustrating because it is not what this competition should be about. As a researcher in the field, I want to see interesting solutions come out of this competition, and these submission problems are reducing the amount of actual contributions instead of people forking the one or two notebooks that do work.
", "votes": 3, "replies": [ { "id": 2286012, "author_name": "something4kag", "author_url": "", "post_date": "2023-06-03T07:07:57.593000", "content": "Figured there would be pushback on this and the petty downvotes. But the point being made was in reference to this post. and this. George Megre and Forrato have been able to make successful submissions over 9 days ago and put explanations in their comments. Both are using pipelines similar to the previous competition and there are code snippets here from them as well as a provided notebook link above. It is possible to consider their example and to look at the discussions to help find solutions. e.g. from Forrato:
\n\"About Conv2D and TFLite you can read in comments to my team's write-up from previous competition.
\nTLDR: not all conv types are good convertable to TFLite with torch-onnx-keras pipeline. But all are convertable with nobuco. But in Kaggle environment some of them slower than our keras implemetation (But locally in different environment nobuco is better). So with nobuco no errors but slower\"
In reference to learning, it is also possible to try working with Tensorflow rather than Pytorch if there are issues with Pytorch or onnx.
\nThe latest post that the competition evaluation is using TFlite runtime 2.14 seems bizarre since that does not appear to be available anywhere. So clarification on where that is to be obtained has been asked here
", "votes": 0, "replies": [] }, { "id": 2286134, "author_name": "Mathieu De Coster", "author_url": "", "post_date": "2023-06-03T08:45:00.090000", "content": "Hey @something4kag
\nI am also using the exact same pipeline from the previous competition which worked there and where we also had to take into account the Conv conversion issues, so the code should not be the issue, the issue is with the environment.
\nI am also, like many others, unable to submit very simple models with Tensorflow code (so no PyTorch conversion).
\nSee for example this post from @markwijkhuizen https://www.kaggle.com/competitions/asl-fingerspelling/discussion/414631 who also got a good score on the previous competition.
\nI've also looked in detail at the posts with solutions you posted, and unfortunately those do not seem to solve my issues. We're not just complaining here, we're trying to find a solution that works for everyone :)
\nFor getting version 2.14, check my reply to your comment. Note that you will get an error if you import both tensorflow and tflite-runtime with version 2.14.
", "votes": 1, "replies": [] }, { "id": 2286205, "author_name": "something4kag", "author_url": "", "post_date": "2023-06-03T09:50:06.733000", "content": "I have added a comment in the post on TFLite Runtime Version number correction
\nhttps://www.kaggle.com/competitions/asl-fingerspelling/discussion/414682
\nfor a way to test your submission.zip with 2.14.0
\nFor the moment it is best to test your model this way, because of the error
\nImportError: generic_type: type \"InterpreterWrapper\" is already registered!
\nif you try to import both Tensorflow and TFLite runtime in the same notebook.
\nIt is a prerelease and dev version so perhaps later will be fixed up. There are ways to do it, but think this is a simple way to get an idea if your model will work or not.