I modified another implementation of CTC and successfully made it kaggle's TPU-compatible.
\nThe original implementation is from here.
\nMy modified TPU-compatible version is here.
\nI modified ROHITH INGILELA's CTC notebook to show how to run it on TPU. My modified version is here.
The original notebook runs in ~6 hours; mine runs in ~47 minutes.
\nEnjoy, and as always- votes would be appreciated :)
I modified another implementation of CTC and successfully made it kaggle's TPU-compatible.
\nThe original implementation is from here.
\nMy modified TPU-compatible version is here.
\nI modified ROHITH INGILELA's CTC notebook to show how to run it on TPU. My modified version is here.
The original notebook runs in ~6 hours; mine runs in ~47 minutes.
\nEnjoy, and as always- votes would be appreciated :)
Maybe I have made some mistakes… But the same model of mine trained on GPU & TPU perform differently, up to 0.04 in score. Wiki tells me that TPU only offer lower precision calculation, this may influence the final performance.
", "rawMarkdown": "Maybe I have made some mistakes... But the same model of mine trained on GPU & TPU perform differently, up to 0.04 in score. Wiki tells me that TPU only offer lower precision calculation, this may influence the final performance.", "votes": 1, "replies": [ { "id": 2358165, "postDate": "2023-07-25T10:31:58.243Z", "content": "Did you use the exact same notebook and the same data and only change TPU to gpu in the console? I.e. If I run my TPU notebook on GPU it will do 0.04 better? If so I will take another look, but is sounds strange. If you refer to a difference from the original score of Rohith's notebook, take a look at the comment section of my notebook and make sure you use version 2 of Rohith's dataset, and not version 6.
", "rawMarkdown": "Did you use the exact same notebook and the same data and only change TPU to gpu in the console? I.e. If I run my TPU notebook on GPU it will do 0.04 better? If so I will take another look, but is sounds strange. If you refer to a difference from the original score of Rohith's notebook, take a look at the comment section of my notebook and make sure you use version 2 of Rohith's dataset, and not version 6." }, { "id": 2374861, "postDate": "2023-08-05T08:58:11.677Z", "content": "Have you solved your problem? I have also encountered a similar problem where the same parameters are significantly lower on TPU by more points.
", "rawMarkdown": "Have you solved your problem? I have also encountered a similar problem where the same parameters are significantly lower on TPU by more points.", "replies": [ { "id": 2375096, "postDate": "2023-08-05T11:52:42.383Z", "content": "He Li had not answered, so I had not looked into this, but since you encountered this problem with the same parameters, I will check it. I hope to have some answers by tomorrow.
", "rawMarkdown": "He Li had not answered, so I had not looked into this, but since you encountered this problem with the same parameters, I will check it. I hope to have some answers by tomorrow." }, { "id": 2375772, "postDate": "2023-08-05T22:05:17.760Z", "content": "/usr/local/lib/python3/dist-packages/keras/utils/generic_utils.py update(self, current, values, finalize)\n )\n avg > :\n--> info += \n :\n info += \n\nValueError: Unknown code of \nWhen I used tf.nn.ctc_loss and bfloat16 in Colab, the issue mentioned above occurred, and I couldn't find a solution for it. Although removing bfloat16 resolved the problem, using it might be necessary to achieve a higher score. In addition to the previously mentioned issue, when I trained the same code without bfloat16 on Kaggle's v3-8, I encountered a problem related to nan loss.
", "rawMarkdown": "```python\n/usr/local/lib/python3.10/dist-packages/keras/utils/generic_utils.py in update(self, current, values, finalize)\n 308 )\n 309 if avg > 1e-3:\n--> 310 info += f\" {avg:.4f}\"\n 311 else:\n 312 info += f\" {avg:.4e}\"\n\nValueError: Unknown format code 'f' for object of type 'str'\n```\n\nWhen I used tf.nn.ctc_loss and bfloat16 in Colab, the issue mentioned above occurred, and I couldn't find a solution for it. Although removing bfloat16 resolved the problem, using it might be necessary to achieve a higher score. In addition to the previously mentioned issue, when I trained the same code without bfloat16 on Kaggle's v3-8, I encountered a problem related to nan loss." }, { "id": 2375824, "postDate": "2023-08-05T23:46:51.543Z", "content": "Ok, I have some answers. After incorporating the latest updates of ROHITH's notebook, my CTC on TPU could not drive the training score as low as Rohith's (3.9321 vs. 3.1931), but achieved a better score by one point on the LB (0.688 vs. 0.687)
", "rawMarkdown": "Ok, I have some answers. After incorporating the latest updates of [ROHITH's notebook](https://www.kaggle.com/code/irohith/aslfr-ctc-based-on-prev-comp-1st-place), my [CTC on TPU](https://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu) could not drive the training score as low as Rohith's (3.9321 vs. 3.1931), [but achieved a better score by one point on the LB](https://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu-submission) (0.688 vs. 0.687)" }, { "id": 2375889, "postDate": "2023-08-06T02:27:46.420Z", "content": "I can't exceed 0.7 points with TPU, it's too difficult to debug.
", "rawMarkdown": "I can't exceed 0.7 points with TPU, it's too difficult to debug." } ] } ] }, { "id": 2356239, "postDate": "2023-07-24T03:57:06.930Z", "content": "Amazing! This should be merged into TensorFlow.
\nI opened an issue here previously: https://github.com/tensorflow/tensorflow/issues/61297#issuecomment-1648022939.
\nWould you like to open a PR to tensorflow to try to add it? @shlomoron
It would probably be redundant, as TensorFlow already has a working version. It just doesn't work on Kaggle, haha. They need to figure out the problem (well, I have a hunch that the problem is on Kaggle's side- something in the environment, probably. The TensorFlow version actually does not fail only on TPU but also on GPU in XLA mode (jit_compile = True), and although I did not check, I guess that it WILL work in, say, colab, considering that it works there on TPU.
", "rawMarkdown": "It would probably be redundant, as TensorFlow already has a working version. It just doesn't work on Kaggle, haha. They need to figure out the problem (well, I have a hunch that the problem is on Kaggle's side- something in the environment, probably. The TensorFlow version actually does not fail only on TPU but also on GPU in XLA mode (jit_compile = True), and although I did not check, I guess that it WILL work in, say, colab, considering that it works there on TPU." } ] }, { "id": 2374282, "postDate": "2023-08-04T20:14:35.100Z", "content": "I am struggling to use the CTC loss with sparse labels (on GPU). The computational speed for that version is from 2 to 10x faster than using dense labels. However, my training always diverge.. I found other people having the same issues online but couldn't find a solution. Does any1 have any trick to solve this?
", "rawMarkdown": "I am struggling to use the CTC loss with sparse labels (on GPU). The computational speed for that version is from 2 to 10x faster than using dense labels. However, my training always diverge.. I found other people having the same issues online but couldn't find a solution. Does any1 have any trick to solve this?" }, { "id": 2356046, "postDate": "2023-07-23T20:29:43.710Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2358055, "author_name": "He Li", "author_url": "", "post_date": "2023-07-25T09:19:26.677000", "content": "Maybe I have made some mistakes… But the same model of mine trained on GPU & TPU perform differently, up to 0.04 in score. Wiki tells me that TPU only offer lower precision calculation, this may influence the final performance.
", "votes": 1, "replies": [ { "id": 2358165, "author_name": "greySnow", "author_url": "", "post_date": "2023-07-25T10:31:58.243000", "content": "Did you use the exact same notebook and the same data and only change TPU to gpu in the console? I.e. If I run my TPU notebook on GPU it will do 0.04 better? If so I will take another look, but is sounds strange. If you refer to a difference from the original score of Rohith's notebook, take a look at the comment section of my notebook and make sure you use version 2 of Rohith's dataset, and not version 6.
", "votes": 0, "replies": [] }, { "id": 2374861, "author_name": "Scenery SunFireInk", "author_url": "", "post_date": "2023-08-05T08:58:11.677000", "content": "Have you solved your problem? I have also encountered a similar problem where the same parameters are significantly lower on TPU by more points.
", "votes": 0, "replies": [ { "id": 2375096, "author_name": "greySnow", "author_url": "", "post_date": "2023-08-05T11:52:42.383000", "content": "He Li had not answered, so I had not looked into this, but since you encountered this problem with the same parameters, I will check it. I hope to have some answers by tomorrow.
", "votes": 0, "replies": [] }, { "id": 2375772, "author_name": "Scenery SunFireInk", "author_url": "", "post_date": "2023-08-05T22:05:17.760000", "content": "/usr/local/lib/python3/dist-packages/keras/utils/generic_utils.py update(self, current, values, finalize)\n )\n avg > :\n--> info += \n :\n info += \n\nValueError: Unknown code of \nWhen I used tf.nn.ctc_loss and bfloat16 in Colab, the issue mentioned above occurred, and I couldn't find a solution for it. Although removing bfloat16 resolved the problem, using it might be necessary to achieve a higher score. In addition to the previously mentioned issue, when I trained the same code without bfloat16 on Kaggle's v3-8, I encountered a problem related to nan loss.
", "votes": 0, "replies": [] }, { "id": 2375824, "author_name": "greySnow", "author_url": "", "post_date": "2023-08-05T23:46:51.543000", "content": "Ok, I have some answers. After incorporating the latest updates of ROHITH's notebook, my CTC on TPU could not drive the training score as low as Rohith's (3.9321 vs. 3.1931), but achieved a better score by one point on the LB (0.688 vs. 0.687)
", "votes": 0, "replies": [] }, { "id": 2375889, "author_name": "Scenery SunFireInk", "author_url": "", "post_date": "2023-08-06T02:27:46.420000", "content": "I can't exceed 0.7 points with TPU, it's too difficult to debug.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2356239, "author_name": "WalkingMoose", "author_url": "", "post_date": "2023-07-24T03:57:06.930000", "content": "Amazing! This should be merged into TensorFlow.
\nI opened an issue here previously: https://github.com/tensorflow/tensorflow/issues/61297#issuecomment-1648022939.
\nWould you like to open a PR to tensorflow to try to add it? @shlomoron
It would probably be redundant, as TensorFlow already has a working version. It just doesn't work on Kaggle, haha. They need to figure out the problem (well, I have a hunch that the problem is on Kaggle's side- something in the environment, probably. The TensorFlow version actually does not fail only on TPU but also on GPU in XLA mode (jit_compile = True), and although I did not check, I guess that it WILL work in, say, colab, considering that it works there on TPU.
", "votes": 0, "replies": [] } ] }, { "id": 2374282, "author_name": "Adriano Passos", "author_url": "", "post_date": "2023-08-04T20:14:35.100000", "content": "I am struggling to use the CTC loss with sparse labels (on GPU). The computational speed for that version is from 2 to 10x faster than using dense labels. However, my training always diverge.. I found other people having the same issues online but couldn't find a solution. Does any1 have any trick to solve this?
", "votes": 0, "replies": [] }, { "id": 2356046, "author_name": "", "author_url": "", "post_date": "2023-07-23T20:29:43.710000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2356040": "I modified another implementation of CTC and successfully made it kaggle's TPU-compatible.\nThe original implementation [is from here](https://github.com/alexeytochin/tf_seq2seq_losses).\nMy modified TPU-compatible version [is here](https://www.kaggle.com/datasets/shlomoron/ctc-tpu).\nI modified [ROHITH INGILELA's CTC notebook](https://www.kaggle.com/code/irohith/aslfr-ctc-based-on-prev-comp-1st-place) to show how to run it on TPU. My modified version [is here](https://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu).\n\nThe original notebook runs in ~6 hours; mine runs in ~47 minutes.\nEnjoy, and as always- votes would be appreciated :)", "2358055": "Maybe I have made some mistakes... But the same model of mine trained on GPU & TPU perform differently, up to 0.04 in score. Wiki tells me that TPU only offer lower precision calculation, this may influence the final performance.", "2356239": "Amazing! This should be merged into TensorFlow. \nI opened an issue here previously: https://github.com/tensorflow/tensorflow/issues/61297#issuecomment-1648022939.\nWould you like to open a PR to tensorflow to try to add it? @shlomoron ", "2374282": "I am struggling to use the CTC loss with sparse labels (on GPU). The computational speed for that version is from 2 to 10x faster than using dense labels. However, my training always diverge.. I found other people having the same issues online but couldn't find a solution. Does any1 have any trick to solve this?", "2356046": "" } }