Due to an internal miscommunication the metric has been using TF Lite runtime version 2.14.0 rather than 2.9.1 as previously stated on the evaluation tab. We are going to stick with version 2.14.0 to avoid breaking submissions that have already succeeded.
\nI don't know how many failing submissions were caused by this issue, but I really apologize to everyone affected by this.
\nEdit: here is the exact TF Lite runtime wheel we've been using.
", "messages": [ { "id": 2285407, "postDate": "2023-06-02T17:27:23.427Z", "content": "Due to an internal miscommunication the metric has been using TF Lite runtime version 2.14.0 rather than 2.9.1 as previously stated on the evaluation tab. We are going to stick with version 2.14.0 to avoid breaking submissions that have already succeeded.
\nI don't know how many failing submissions were caused by this issue, but I really apologize to everyone affected by this.
\nEdit: here is the exact TF Lite runtime wheel we've been using.
", "rawMarkdown": "Due to an internal miscommunication the metric has been using TF Lite runtime version 2.14.0 rather than 2.9.1 as previously stated on the evaluation tab. We are going to stick with version 2.14.0 to avoid breaking submissions that have already succeeded.\n\nI don't know how many failing submissions were caused by this issue, but I really apologize to everyone affected by this.\n\nEdit: here is [the exact TF Lite runtime wheel we've been using](https://www.kaggle.com/datasets/philculliton/tflite-wheels-2140). ", "votes": 14 }, { "id": 2291745, "postDate": "2023-06-07T19:40:21.450Z", "content": "I've done some more digging and wanted to provide the full context of how we got here and why I'm generally inclined to stick with the existing setup. Kaggle notebooks upgraded to Python 3.10 shortly before this competition was launched. Prior to that, our drafts had indeed been using TF Lite 2.9.1. It looks like the first TF Lite release to support Python 3.10 was 2.13.0, which is also not yet a stable branch. If we were to migrate to a stable branch we would also need to use an old Python 3.7 notebook environment and would roll all the way back to TF Lite 2.9.1.
", "rawMarkdown": "I've done some more digging and wanted to provide the full context of how we got here and why I'm generally inclined to stick with the existing setup. Kaggle notebooks upgraded to Python 3.10 shortly before this competition was launched. Prior to that, our drafts had indeed been using TF Lite 2.9.1. It looks like the first TF Lite release to support Python 3.10 [was 2.13.0](https://github.com/tensorflow/tensorflow/commit/62d4b1627c3e54e445248178214ef469c6b65981), which is also not yet a stable branch. If we were to migrate to a stable branch we would also need to use an old Python 3.7 notebook environment and would roll all the way back to TF Lite 2.9.1.", "votes": 5, "replies": [ { "id": 2291874, "postDate": "2023-06-07T22:29:08.097Z", "content": "This makes sense! Do you know which nightly is being used, just so we're all on the same page?
", "rawMarkdown": "This makes sense! Do you know which nightly is being used, just so we're all on the same page?", "replies": [ { "id": 2291897, "postDate": "2023-06-07T23:24:39.683Z", "content": "", "rawMarkdown": "[2.14.0.dev20230508-cp310](https://www.kaggle.com/datasets/philculliton/tflite-wheels-2140)", "votes": 3 }, { "id": 2291911, "postDate": "2023-06-08T00:02:57.503Z", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!" } ] } ] }, { "id": 2286513, "postDate": "2023-06-03T14:18:43.083Z", "content": "The tflite-runtime-nightly pip repository lists a few dozen versions, which specific version is being used for inference?
\nCould you also elaborate on the decision to use an unstable nightly build for inference?
\nGiven the major issues with inference the latest stable version, 2.12, might have caused less issues.
", "rawMarkdown": "The [tflite-runtime-nightly pip repository](https://pypi.org/project/tflite-runtime-nightly/#history) lists a few dozen versions, which specific version is being used for inference?\n\nCould you also elaborate on the decision to use an **unstable** nightly build for inference?\n\nGiven the major issues with inference the latest stable version, 2.12, might have caused less issues.", "votes": 5 }, { "id": 2286196, "postDate": "2023-06-03T09:42:51.497Z", "content": "FYI - I created a dataset https://www.kaggle.com/datasets/something4kag/tflite-runtime-nightly-2140dev20230602-whl
\nfor the latest prerelease tflite-runtime-nightly 2.14.0.dev…
\nIf you do pip install pip install tflite-runtime-nightly you will get tflite_runtime_nightly-2.13.0.dev…
I have also created a notebook using TFlite runtime 2.14.0 to test the submission.zip from irohith's notebook output.
\nhttps://www.kaggle.com/code/something4kag/aslfr-transformer-irohith-tflite-runtime-2-14-0/notebook
This notebook uses Python 3.10 and TFlite runtime 2.14.0 and can be adapted to test your own submission.zip to ensure it works.
\nMany of the notebooks test using the TFliteConverter which may account for why they seem to work OK.
Good Luck!
\nEdit - have added 2 notebooks to show a preprocessing example to remove the NaN rows and hands as I commented in the discussions, and a notebook to test the submission.zip created with tflite-runtime-nightly-2.14.0. Then if successful to do a submission which was fine.
\nhttps://www.kaggle.com/code/something4kag/asl-f-preprocessing-example
\nhttps://www.kaggle.com/code/something4kag/asl-f-preprocessing-example-submission-test
Hi @Sohier,
\nThank you for confirming our suspicions that the scoring pipeline TfLite version did not match the instructions. However, after checking, I found that there are 25 releases of TfLite-runtime 2.14. In addition, this is a \"nightly\" version and therefore not stable. If the version was frozen at the start of the competition, it probably still contained bugs, since new versions have being released almost daily in May. This could explain why, for example, there have been extensive problems to port Pytorch models to TfLite (problems that did not occur in the previous competition).
\nI really don't think that using an unstable experimental version is giving all participants a fair chance in this respect!!
For this reason,I would strongly plead to reverse the scoring pipeline to the last stable version of TfLite (i.e. 2.12). Given that most of us are still struggling to get pre-processing implemented (I'm not even sure whether ANYONE has in fact been able to add all the preprocessing they would like to add), I don't expect there would be a lot of invalidated solutions …
\nIf that is impossible (which I would strongly regret), then please al least let us know exactly which release of TfLite 2.14-runtime is being used (or maybe move forward to the very latest one in which a few bugs have hopefully already been fixed)?
Also, since almost all of us (except for a few) have now lost almost 3 weeks of time because of this, I would also like to ask whether it could be considered to extend the deadline to compensate for this.
", "rawMarkdown": "Hi @Sohier,\n\nThank you for confirming our suspicions that the scoring pipeline TfLite version did not match the instructions. However, after checking, I found that there are 25 releases of TfLite-runtime 2.14. In addition, this is a \"nightly\" version and therefore not stable. If the version was frozen at the start of the competition, it probably still contained bugs, since new versions have being released almost daily in May. This could explain why, for example, there have been extensive problems to port Pytorch models to TfLite (problems that did not occur in the previous competition). \nI really don't think that using an unstable experimental version is giving all participants a fair chance in this respect!!\n\nFor this reason,**I would strongly plead to reverse the scoring pipeline to the last stable version of TfLite (i.e. 2.12).** Given that most of us are still struggling to get pre-processing implemented (I'm not even sure whether ANYONE has in fact been able to add all the preprocessing they would like to add), I don't expect there would be a lot of invalidated solutions ...\nIf that is impossible (which I would strongly regret), then please al least let us know exactly which release of TfLite 2.14-runtime is being used (or maybe move forward to the very latest one in which a few bugs have hopefully already been fixed)?\n\nAlso, since almost all of us (except for a few) have now lost almost 3 weeks of time because of this, I would also like to ask whether it could be considered to extend the deadline to compensate for this.\n\n", "votes": 3, "replies": [ { "id": 2286170, "postDate": "2023-06-03T09:20:55.380Z", "content": "The competition launched on May 11th, I have tried all versions of tflite-runtime-nightly==2.14.0.dev202305XY with XY from 03 to 10, which I assume is the latest version that could have been used if it had been fixed at the start of this competition (11 doesn't exist).
I took a notebook which I know fails the scoring step, but ran locally with older TFLite runtimes, and ran it on the entire training set for all of these versions. It ran locally on all of them without any errors. Just to be sure, I did another submision with the same notebook, and sure enough it did not pass scoring again.
\nTaking into account this comment about dimension mismatches, I think the TFLite runtime version mismatch is not causing the failures in the submission pipeline, but perhaps some edge cases in the test set that do not occur in the training set, with certain TensorFlow operations. It is interesting that the shape mismatches appear to be off-by-one errors.
", "rawMarkdown": "The competition launched on May 11th, I have tried all versions of `tflite-runtime-nightly==2.14.0.dev202305XY` with `XY` from `03` to `10`, which I assume is the latest version that could have been used if it had been fixed at the start of this competition (`11` doesn't exist).\n\nI took a notebook which I know fails the scoring step, but ran locally with older TFLite runtimes, and ran it on the entire training set for all of these versions. It ran locally on all of them without any errors. Just to be sure, I did another submision with the same notebook, and sure enough it did not pass scoring again.\n\nTaking into account [this comment about dimension mismatches](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/413598#2285409), I think the TFLite runtime version mismatch is not causing the failures in the submission pipeline, but perhaps some edge cases in the test set that do not occur in the training set, with certain TensorFlow operations. It is interesting that the shape mismatches appear to be off-by-one errors.", "votes": 4 } ] }, { "id": 2289864, "postDate": "2023-06-06T11:42:46.690Z", "content": "Hi @sohier ,
\nAlthough more and more people are submitting working models, we would still really appreciate it if you could let us know exactly which release of tflite-runtime-nightly v2.14 is used in the scoring pipeline??
\nAnd also, I would like to repeat my question whether it would be possible to shift the deadline with a few weeks to compensate for the lost time at the start of the competition?
", "rawMarkdown": "Hi @sohier ,\n\nAlthough more and more people are submitting working models, we would still really appreciate it if you could let us know exactly which release of tflite-runtime-nightly v2.14 is used in the scoring pipeline??\n\nAnd also, I would like to repeat my question whether it would be possible to shift the deadline with a few weeks to compensate for the lost time at the start of the competition?", "votes": 2, "replies": [ { "id": 2294163, "postDate": "2023-06-09T20:33:01.090Z", "content": "We've gone ahead with a deadline extension: https://www.kaggle.com/competitions/asl-fingerspelling/discussion/416149
\nThank you for advocating for that change!
", "rawMarkdown": "We've gone ahead with a deadline extension: https://www.kaggle.com/competitions/asl-fingerspelling/discussion/416149\n\nThank you for advocating for that change!", "votes": 1, "replies": [ { "id": 2294714, "postDate": "2023-06-10T08:59:01.017Z", "content": "Thank you, that's wonderful. This will allow many people to reschedule their efforts around already planned holidays.
\nI am very happy that all issues have now been cleared out and I hope that, as a Kaggle community, we will be able to improve the state of the art for continuous fingerspelling recognition (as we already have for isolated sign recognition) and pave the way for AI to open up a more inclusive society.
", "rawMarkdown": "Thank you, that's wonderful. This will allow many people to reschedule their efforts around already planned holidays.\n\nI am very happy that all issues have now been cleared out and I hope that, as a Kaggle community, we will be able to improve the state of the art for continuous fingerspelling recognition (as we already have for isolated sign recognition) and pave the way for AI to open up a more inclusive society." } ] } ] }, { "id": 2354461, "postDate": "2023-07-22T13:20:24.197Z", "content": "Hi,
\nDoes the current environment being used for evaluation allow TFOps for tflite? (which can be set using tf.lite.OpsSet.SELECT_TF_OPS)
\nAny guidance on this would be greatly appreciated because it would make dealing with variable strings far easier and allow people to build potentially models with faster throughputs :).
I'm having some issues with importing the runtime, I imported the dataset linked in the post and install it by path. if anyone could give me some help I'd appreciate it:
\n\n\npip install \"path to wheel\"
\n
\n import tflite_runtime.interpreter as tflite
The error is:
\n\n\nImportError: generic_type: type \"InterpreterWrapper\" is already registered!
\n
This did not appear when using python 3.7 with tflite 2.9.1 so I'm not really sure why this happens. Thanks!
", "rawMarkdown": "I'm having some issues with importing the runtime, I imported the dataset linked in the post and install it by path. if anyone could give me some help I'd appreciate it:\n\n>pip install \"path to wheel\"\n>import tflite_runtime.interpreter as tflite\n\nThe error is:\n\n>ImportError: generic_type: type \"InterpreterWrapper\" is already registered!\n\nThis did not appear when using python 3.7 with tflite 2.9.1 so I'm not really sure why this happens. Thanks!", "replies": [ { "id": 2313010, "postDate": "2023-06-22T10:47:18.720Z", "content": "I have faced same issue. But if you just \"Copy and Edit\" this notebook everything is fine for tests
\nI guess you should not import tensorflow before it. Because if you firstly import tflite_runtime you'll get an error while import tensorflow
\nBut actually no any differences with python 3.7, tflite 2.9.1 version. If it successful in both versions, it still can get error while submission
", "rawMarkdown": "I have faced same issue. But if you just \"Copy and Edit\" this [notebook](https://www.kaggle.com/code/something4kag/asl-f-preprocessing-example-submission-test) everything is fine for tests\n\nI guess you should not import tensorflow before it. Because if you firstly import tflite_runtime you'll get an error while import tensorflow\n\nBut actually no any differences with python 3.7, tflite 2.9.1 version. If it successful in both versions, it still can get error while submission", "votes": 1 } ] }, { "id": 2309847, "postDate": "2023-06-19T23:45:25.970Z", "content": "Hi @sohier , I have a little question about the contest submission, please help me answer it.
\nThe notebook, https://www.kaggle.com/code/finlay/aslfr-transformer-submit/, use tflite model from other notebook and submit, the score is ok. Is this submission method finally allowed?
", "rawMarkdown": "Hi @sohier , I have a little question about the contest submission, please help me answer it.\n\nThe notebook, https://www.kaggle.com/code/finlay/aslfr-transformer-submit/, use tflite model from other notebook and submit, the score is ok. Is this submission method finally allowed?\n", "replies": [ { "id": 2310153, "postDate": "2023-06-20T06:38:51.203Z", "content": "Hi Finlay,
\nYes, in this competition you only have to submit the TfLite model. Many (most?) of us are training offline, uploading our models or .zip files and using a submit-only notebook.
\nIf however, you would end up in the prizes range, you will be expected to provide all your code so the organizers can check that you followed the competition rules (e.g., w.r.t. external data).
", "rawMarkdown": "Hi Finlay,\n\nYes, in this competition you only have to submit the TfLite model. Many (most?) of us are training offline, uploading our models or .zip files and using a submit-only notebook.\n\nIf however, you would end up in the prizes range, you will be expected to provide all your code so the organizers can check that you followed the competition rules (e.g., w.r.t. external data).", "votes": 1 } ] }, { "id": 2293750, "postDate": "2023-06-09T13:00:46.763Z", "content": "Thank you for sharing, the comments you made are very informative.
", "rawMarkdown": "Thank you for sharing, the comments you made are very informative." }, { "id": 2285992, "postDate": "2023-06-03T06:34:55.047Z", "content": "", "rawMarkdown": "~~Just curious how you have TF Lite runtime version 2.14.0 \nThe latest on https://pypi.org/project/tflite-runtime/#history is tflite-runtime 2.12.0 Released: May 9, 2023\ncompatible with Python 3.7, 3.8, 3.9. Since notebooks created now use Python 3.10 install latest has not been an option.\nSo where does one get 2.14.0 and is it compatible with Python 3.10 ?? ~~", "replies": [ { "id": 2286128, "postDate": "2023-06-03T08:39:28.020Z", "content": "You can use pip install tflite-runtime-nightly to get the 2.14 version, which appears to still be in development.
See post above for how to on 2.14.0 if needed.
", "rawMarkdown": "See post above for how to on 2.14.0 if needed." } ] }, { "id": 2285726, "postDate": "2023-06-03T00:04:33.017Z", "content": "Thanks for looking into this and figuring it out, Sohier!
\nAs @wonderingalice says, it would be good to figure out exactly what subversion is being used and whether it has any significant bugs
", "rawMarkdown": "Thanks for looking into this and figuring it out, Sohier!\n\nAs @wonderingalice says, it would be good to figure out exactly what subversion is being used and whether it has any significant bugs" }, { "id": 2285437, "postDate": "2023-06-02T17:51:48.080Z", "content": "Thanks for checking and for letting us know @sohier!
", "rawMarkdown": "Thanks for checking and for letting us know @sohier!" }, { "id": 2296343, "postDate": "2023-06-11T18:22:55.827Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2294138, "postDate": "2023-06-09T19:33:37.617Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 2287525, "postDate": "2023-06-04T15:43:14.743Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2291745, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-06-07T19:40:21.450000", "content": "I've done some more digging and wanted to provide the full context of how we got here and why I'm generally inclined to stick with the existing setup. Kaggle notebooks upgraded to Python 3.10 shortly before this competition was launched. Prior to that, our drafts had indeed been using TF Lite 2.9.1. It looks like the first TF Lite release to support Python 3.10 was 2.13.0, which is also not yet a stable branch. If we were to migrate to a stable branch we would also need to use an old Python 3.7 notebook environment and would roll all the way back to TF Lite 2.9.1.
", "votes": 5, "replies": [ { "id": 2291874, "author_name": "anokas", "author_url": "", "post_date": "2023-06-07T22:29:08.097000", "content": "This makes sense! Do you know which nightly is being used, just so we're all on the same page?
", "votes": 0, "replies": [ { "id": 2291897, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-06-07T23:24:39.683000", "content": "", "votes": 3, "replies": [] }, { "id": 2291911, "author_name": "anokas", "author_url": "", "post_date": "2023-06-08T00:02:57.503000", "content": "Thanks a lot!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2286513, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2023-06-03T14:18:43.083000", "content": "The tflite-runtime-nightly pip repository lists a few dozen versions, which specific version is being used for inference?
\nCould you also elaborate on the decision to use an unstable nightly build for inference?
\nGiven the major issues with inference the latest stable version, 2.12, might have caused less issues.
", "votes": 5, "replies": [] }, { "id": 2286196, "author_name": "something4kag", "author_url": "", "post_date": "2023-06-03T09:42:51.497000", "content": "FYI - I created a dataset https://www.kaggle.com/datasets/something4kag/tflite-runtime-nightly-2140dev20230602-whl
\nfor the latest prerelease tflite-runtime-nightly 2.14.0.dev…
\nIf you do pip install pip install tflite-runtime-nightly you will get tflite_runtime_nightly-2.13.0.dev…
I have also created a notebook using TFlite runtime 2.14.0 to test the submission.zip from irohith's notebook output.
\nhttps://www.kaggle.com/code/something4kag/aslfr-transformer-irohith-tflite-runtime-2-14-0/notebook
This notebook uses Python 3.10 and TFlite runtime 2.14.0 and can be adapted to test your own submission.zip to ensure it works.
\nMany of the notebooks test using the TFliteConverter which may account for why they seem to work OK.
Good Luck!
\nEdit - have added 2 notebooks to show a preprocessing example to remove the NaN rows and hands as I commented in the discussions, and a notebook to test the submission.zip created with tflite-runtime-nightly-2.14.0. Then if successful to do a submission which was fine.
\nhttps://www.kaggle.com/code/something4kag/asl-f-preprocessing-example
\nhttps://www.kaggle.com/code/something4kag/asl-f-preprocessing-example-submission-test
Hi @Sohier,
\nThank you for confirming our suspicions that the scoring pipeline TfLite version did not match the instructions. However, after checking, I found that there are 25 releases of TfLite-runtime 2.14. In addition, this is a \"nightly\" version and therefore not stable. If the version was frozen at the start of the competition, it probably still contained bugs, since new versions have being released almost daily in May. This could explain why, for example, there have been extensive problems to port Pytorch models to TfLite (problems that did not occur in the previous competition).
\nI really don't think that using an unstable experimental version is giving all participants a fair chance in this respect!!
For this reason,I would strongly plead to reverse the scoring pipeline to the last stable version of TfLite (i.e. 2.12). Given that most of us are still struggling to get pre-processing implemented (I'm not even sure whether ANYONE has in fact been able to add all the preprocessing they would like to add), I don't expect there would be a lot of invalidated solutions …
\nIf that is impossible (which I would strongly regret), then please al least let us know exactly which release of TfLite 2.14-runtime is being used (or maybe move forward to the very latest one in which a few bugs have hopefully already been fixed)?
Also, since almost all of us (except for a few) have now lost almost 3 weeks of time because of this, I would also like to ask whether it could be considered to extend the deadline to compensate for this.
", "votes": 3, "replies": [ { "id": 2286170, "author_name": "Mathieu De Coster", "author_url": "", "post_date": "2023-06-03T09:20:55.380000", "content": "The competition launched on May 11th, I have tried all versions of tflite-runtime-nightly==2.14.0.dev202305XY with XY from 03 to 10, which I assume is the latest version that could have been used if it had been fixed at the start of this competition (11 doesn't exist).
I took a notebook which I know fails the scoring step, but ran locally with older TFLite runtimes, and ran it on the entire training set for all of these versions. It ran locally on all of them without any errors. Just to be sure, I did another submision with the same notebook, and sure enough it did not pass scoring again.
\nTaking into account this comment about dimension mismatches, I think the TFLite runtime version mismatch is not causing the failures in the submission pipeline, but perhaps some edge cases in the test set that do not occur in the training set, with certain TensorFlow operations. It is interesting that the shape mismatches appear to be off-by-one errors.
", "votes": 4, "replies": [] } ] }, { "id": 2289864, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-06-06T11:42:46.690000", "content": "Hi @sohier ,
\nAlthough more and more people are submitting working models, we would still really appreciate it if you could let us know exactly which release of tflite-runtime-nightly v2.14 is used in the scoring pipeline??
\nAnd also, I would like to repeat my question whether it would be possible to shift the deadline with a few weeks to compensate for the lost time at the start of the competition?
", "votes": 2, "replies": [ { "id": 2294163, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-06-09T20:33:01.090000", "content": "We've gone ahead with a deadline extension: https://www.kaggle.com/competitions/asl-fingerspelling/discussion/416149
\nThank you for advocating for that change!
", "votes": 1, "replies": [ { "id": 2294714, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-06-10T08:59:01.017000", "content": "Thank you, that's wonderful. This will allow many people to reschedule their efforts around already planned holidays.
\nI am very happy that all issues have now been cleared out and I hope that, as a Kaggle community, we will be able to improve the state of the art for continuous fingerspelling recognition (as we already have for isolated sign recognition) and pave the way for AI to open up a more inclusive society.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2354461, "author_name": "Archon512", "author_url": "", "post_date": "2023-07-22T13:20:24.197000", "content": "Hi,
\nDoes the current environment being used for evaluation allow TFOps for tflite? (which can be set using tf.lite.OpsSet.SELECT_TF_OPS)
\nAny guidance on this would be greatly appreciated because it would make dealing with variable strings far easier and allow people to build potentially models with faster throughputs :).
I'm having some issues with importing the runtime, I imported the dataset linked in the post and install it by path. if anyone could give me some help I'd appreciate it:
\n\n\npip install \"path to wheel\"
\n
\n import tflite_runtime.interpreter as tflite
The error is:
\n\n\nImportError: generic_type: type \"InterpreterWrapper\" is already registered!
\n
This did not appear when using python 3.7 with tflite 2.9.1 so I'm not really sure why this happens. Thanks!
", "votes": 0, "replies": [ { "id": 2313010, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-06-22T10:47:18.720000", "content": "I have faced same issue. But if you just \"Copy and Edit\" this notebook everything is fine for tests
\nI guess you should not import tensorflow before it. Because if you firstly import tflite_runtime you'll get an error while import tensorflow
\nBut actually no any differences with python 3.7, tflite 2.9.1 version. If it successful in both versions, it still can get error while submission
", "votes": 1, "replies": [] } ] }, { "id": 2309847, "author_name": "Finlay", "author_url": "", "post_date": "2023-06-19T23:45:25.970000", "content": "Hi @sohier , I have a little question about the contest submission, please help me answer it.
\nThe notebook, https://www.kaggle.com/code/finlay/aslfr-transformer-submit/, use tflite model from other notebook and submit, the score is ok. Is this submission method finally allowed?
", "votes": 0, "replies": [ { "id": 2310153, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-06-20T06:38:51.203000", "content": "Hi Finlay,
\nYes, in this competition you only have to submit the TfLite model. Many (most?) of us are training offline, uploading our models or .zip files and using a submit-only notebook.
\nIf however, you would end up in the prizes range, you will be expected to provide all your code so the organizers can check that you followed the competition rules (e.g., w.r.t. external data).
", "votes": 1, "replies": [] } ] }, { "id": 2293750, "author_name": "Ethan", "author_url": "", "post_date": "2023-06-09T13:00:46.763000", "content": "Thank you for sharing, the comments you made are very informative.
", "votes": 0, "replies": [] }, { "id": 2285992, "author_name": "something4kag", "author_url": "", "post_date": "2023-06-03T06:34:55.047000", "content": "", "votes": 0, "replies": [ { "id": 2286128, "author_name": "Mathieu De Coster", "author_url": "", "post_date": "2023-06-03T08:39:28.020000", "content": "You can use pip install tflite-runtime-nightly to get the 2.14 version, which appears to still be in development.
See post above for how to on 2.14.0 if needed.
", "votes": 0, "replies": [] } ] }, { "id": 2285726, "author_name": "anokas", "author_url": "", "post_date": "2023-06-03T00:04:33.017000", "content": "Thanks for looking into this and figuring it out, Sohier!
\nAs @wonderingalice says, it would be good to figure out exactly what subversion is being used and whether it has any significant bugs
", "votes": 0, "replies": [] }, { "id": 2285437, "author_name": "Mathieu De Coster", "author_url": "", "post_date": "2023-06-02T17:51:48.080000", "content": "Thanks for checking and for letting us know @sohier!
", "votes": 0, "replies": [] }, { "id": 2296343, "author_name": "", "author_url": "", "post_date": "2023-06-11T18:22:55.827000", "content": "", "votes": 0, "replies": [] }, { "id": 2294138, "author_name": "", "author_url": "", "post_date": "2023-06-09T19:33:37.617000", "content": "", "votes": 1, "replies": [] }, { "id": 2287525, "author_name": "", "author_url": "", "post_date": "2023-06-04T15:43:14.743000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2285407": "Due to an internal miscommunication the metric has been using TF Lite runtime version 2.14.0 rather than 2.9.1 as previously stated on the evaluation tab. We are going to stick with version 2.14.0 to avoid breaking submissions that have already succeeded.\n\nI don't know how many failing submissions were caused by this issue, but I really apologize to everyone affected by this.\n\nEdit: here is [the exact TF Lite runtime wheel we've been using](https://www.kaggle.com/datasets/philculliton/tflite-wheels-2140). ", "2291745": "I've done some more digging and wanted to provide the full context of how we got here and why I'm generally inclined to stick with the existing setup. Kaggle notebooks upgraded to Python 3.10 shortly before this competition was launched. Prior to that, our drafts had indeed been using TF Lite 2.9.1. It looks like the first TF Lite release to support Python 3.10 [was 2.13.0](https://github.com/tensorflow/tensorflow/commit/62d4b1627c3e54e445248178214ef469c6b65981), which is also not yet a stable branch. If we were to migrate to a stable branch we would also need to use an old Python 3.7 notebook environment and would roll all the way back to TF Lite 2.9.1.", "2286513": "The [tflite-runtime-nightly pip repository](https://pypi.org/project/tflite-runtime-nightly/#history) lists a few dozen versions, which specific version is being used for inference?\n\nCould you also elaborate on the decision to use an **unstable** nightly build for inference?\n\nGiven the major issues with inference the latest stable version, 2.12, might have caused less issues.", "2286196": "FYI - I created a dataset https://www.kaggle.com/datasets/something4kag/tflite-runtime-nightly-2140dev20230602-whl\nfor the latest prerelease tflite-runtime-nightly 2.14.0.dev... \nIf you do pip install pip install tflite-runtime-nightly you will get tflite_runtime_nightly-2.13.0.dev... \n\nI have also created a notebook using TFlite runtime 2.14.0 to test the submission.zip from irohith's notebook output.\nhttps://www.kaggle.com/code/something4kag/aslfr-transformer-irohith-tflite-runtime-2-14-0/notebook\n\nThis notebook uses Python 3.10 and TFlite runtime 2.14.0 and can be adapted to test your own submission.zip to ensure it works.\nMany of the notebooks test using the TFliteConverter which may account for why they seem to work OK.\n\nGood Luck!\n\nEdit - have added 2 notebooks to show a preprocessing example to remove the NaN rows and hands as I commented in the discussions, and a notebook to test the submission.zip created with tflite-runtime-nightly-2.14.0. Then if successful to do a submission which was fine.\n\nhttps://www.kaggle.com/code/something4kag/asl-f-preprocessing-example\nhttps://www.kaggle.com/code/something4kag/asl-f-preprocessing-example-submission-test", "2285590": "Hi @Sohier,\n\nThank you for confirming our suspicions that the scoring pipeline TfLite version did not match the instructions. However, after checking, I found that there are 25 releases of TfLite-runtime 2.14. In addition, this is a \"nightly\" version and therefore not stable. If the version was frozen at the start of the competition, it probably still contained bugs, since new versions have being released almost daily in May. This could explain why, for example, there have been extensive problems to port Pytorch models to TfLite (problems that did not occur in the previous competition). \nI really don't think that using an unstable experimental version is giving all participants a fair chance in this respect!!\n\nFor this reason,**I would strongly plead to reverse the scoring pipeline to the last stable version of TfLite (i.e. 2.12).** Given that most of us are still struggling to get pre-processing implemented (I'm not even sure whether ANYONE has in fact been able to add all the preprocessing they would like to add), I don't expect there would be a lot of invalidated solutions ...\nIf that is impossible (which I would strongly regret), then please al least let us know exactly which release of TfLite 2.14-runtime is being used (or maybe move forward to the very latest one in which a few bugs have hopefully already been fixed)?\n\nAlso, since almost all of us (except for a few) have now lost almost 3 weeks of time because of this, I would also like to ask whether it could be considered to extend the deadline to compensate for this.\n\n", "2289864": "Hi @sohier ,\n\nAlthough more and more people are submitting working models, we would still really appreciate it if you could let us know exactly which release of tflite-runtime-nightly v2.14 is used in the scoring pipeline??\n\nAnd also, I would like to repeat my question whether it would be possible to shift the deadline with a few weeks to compensate for the lost time at the start of the competition?", "2354461": "Hi,\nDoes the current environment being used for evaluation allow TFOps for tflite? (which can be set using tf.lite.OpsSet.SELECT_TF_OPS)\nAny guidance on this would be greatly appreciated because it would make dealing with variable strings far easier and allow people to build potentially models with faster throughputs :).", "2312987": "I'm having some issues with importing the runtime, I imported the dataset linked in the post and install it by path. if anyone could give me some help I'd appreciate it:\n\n>pip install \"path to wheel\"\n>import tflite_runtime.interpreter as tflite\n\nThe error is:\n\n>ImportError: generic_type: type \"InterpreterWrapper\" is already registered!\n\nThis did not appear when using python 3.7 with tflite 2.9.1 so I'm not really sure why this happens. Thanks!", "2309847": "Hi @sohier , I have a little question about the contest submission, please help me answer it.\n\nThe notebook, https://www.kaggle.com/code/finlay/aslfr-transformer-submit/, use tflite model from other notebook and submit, the score is ok. Is this submission method finally allowed?\n", "2293750": "Thank you for sharing, the comments you made are very informative.", "2285992": "~~Just curious how you have TF Lite runtime version 2.14.0 \nThe latest on https://pypi.org/project/tflite-runtime/#history is tflite-runtime 2.12.0 Released: May 9, 2023\ncompatible with Python 3.7, 3.8, 3.9. Since notebooks created now use Python 3.10 install latest has not been an option.\nSo where does one get 2.14.0 and is it compatible with Python 3.10 ?? ~~", "2285726": "Thanks for looking into this and figuring it out, Sohier!\n\nAs @wonderingalice says, it would be good to figure out exactly what subversion is being used and whether it has any significant bugs", "2285437": "Thanks for checking and for letting us know @sohier!", "2296343": "", "2294138": "", "2287525": "" } }