![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd0fad99126ea85590a0d49dadff5b83a%2FSelection_999(2958).png?generation=1692928740749828&alt=media\")
i think many kagglers will reveal their trick later, but here is mine
\nA. label smooth ====
\nbecause label smooth limits the max probability, it forces the training to get more correct (instead of few very correct).
\n total loss = *smooth_kl_loss + (-)*CTC loss \n\"a\" is scheduled from e.g. 0.01, 0.005, 0.001 and finally to be zero at the end.
\ni am more interested in the following two points:
\nhow we know that long epoch training is a method
\n(for me it is a bug that makes my training overnight and run longer then intended)
why?
\nis it because of the blank token prediction (which is essentially the majority class)
what are your take on the last two points? thanks!
", "messages": [ { "id": 2407264, "postDate": "2023-08-25T01:59:09.017Z", "content": "i think many kagglers will reveal their trick later, but here is mine
\nA. label smooth ====
\nbecause label smooth limits the max probability, it forces the training to get more correct (instead of few very correct).
\n total loss = *smooth_kl_loss + (-)*CTC loss \n\"a\" is scheduled from e.g. 0.01, 0.005, 0.001 and finally to be zero at the end.
\ni am more interested in the following two points:
\nhow we know that long epoch training is a method
\n(for me it is a bug that makes my training overnight and run longer then intended)
why?
\nis it because of the blank token prediction (which is essentially the majority class)
what are your take on the last two points? thanks!
", "rawMarkdown": "i think many kagglers will reveal their trick later, but here is mine\n\nA. label smooth ====\n1. add label smooth (using kl div loss with uniform prior)\n(see this https://discuss.pytorch.org/t/label-smoothing-with-ctcloss/103392. do read the paper!)\n\nbecause label smooth limits the max probability, it forces the training to get more correct (instead of few very correct).\n\n2. slowly remove label smooth loss at the end: i.e.\n```\n total loss = a*smooth_kl_loss + (1-a)*CTC loss \n```\n\n\"a\" is scheduled from e.g. 0.01, 0.005, 0.001 and finally to be zero at the end.\n\n----\n\ni am more interested in the following two points:\n\n- how we know that long epoch training is a method\n(for me it is a bug that makes my training overnight and run longer then intended)\n\n- why? \nis it because of the blank token prediction (which is essentially the majority class)\n\n.png?generation=1692928740749828&alt=media)\n\n----\n\nwhat are your take on the last two points? thanks!\n", "votes": 5 }, { "id": 2407278, "postDate": "2023-08-25T02:21:32.563Z", "content": "how prediction actually looks like:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fa85cc9582722d3cce00a4b5413314c0c%2FSelection_999(2954).png?generation=1692929804073022&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F419af5e2c33e56b307d6f07619a078b0%2FSelection_999(2955).png?generation=1692929824423460&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fcbda4b7cd6e0e91b0b8a75522dbf39b9%2FSelection_999(2953).png?generation=1692930446718283&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fff3411a10b87fd9ba41008e9201c2b2d%2FSelection_999(2952).png?generation=1692930844104357&alt=media\")
red: rnnt
\ngreen:ctc
both used same shared encoder (pretraind with ctc).. only the decoder heads changed. both decoder head had 0.795 in local CV.
\nthe red rnnt prediction is a bit blur because it uses encoder features at dim x half T (for inference sppedup).
\ngreen ctc uses dim x T. (i use T=256 frame length)
you can see EOS prediction for red rnnt (at the bottom). no PAD prediction for rnnt becuase it uses EOS termination.
\nyou can see BLANK,PAD oscillation for green CTC at the end.
the long yellow line is the BLANK prediction
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe8ee605aeec932216bf564008ed8819e%2FSelection_999(2950).png?generation=1692929841114185&alt=media\")
plot of RNNT and CTC scores
\ni estimate upper bound for ensemble : max(RNNT score, CTC score)=0.808. an improvement of 0.01. but i cannot find a way that is simple enough to fit into tflite, so i did not try and submit. (i only estimate the upper bound).
\nbasically you need to first find the non-blank time interval first (alignment), then for that alignment, average model results.
\nit is like detection ensemble, find box cluster and average
", "rawMarkdown": "how prediction actually looks like:\n\n.png?generation=1692929804073022&alt=media)\n\n.png?generation=1692929824423460&alt=media)\n\n.png?generation=1692930446718283&alt=media)\n\n\n.png?generation=1692930844104357&alt=media)\n\n\nred: rnnt\ngreen:ctc\n\nboth used same shared encoder (pretraind with ctc).. only the decoder heads changed. both decoder head had 0.795 in local CV.\nthe red rnnt prediction is a bit blur because it uses encoder features at dim x half T (for inference sppedup).\ngreen ctc uses dim x T. (i use T=256 frame length)\n\nyou can see EOS prediction for red rnnt (at the bottom). no PAD prediction for rnnt becuase it uses EOS termination.\nyou can see BLANK,PAD oscillation for green CTC at the end.\n\nthe long yellow line is the BLANK prediction \n\n\n----\n\n\n\n.png?generation=1692929841114185&alt=media)\n\nplot of RNNT and CTC scores\n\ni estimate upper bound for ensemble : max(RNNT score, CTC score)=0.808. an improvement of 0.01. but i cannot find a way that is simple enough to fit into tflite, so i did not try and submit. (i only estimate the upper bound).\n\nbasically you need to first find the non-blank time interval first (alignment), then for that alignment, average model results.\n\nit is like detection ensemble, find box cluster and average ", "votes": 1, "replies": [ { "id": 2407290, "postDate": "2023-08-25T02:36:22.860Z", "content": "code to compute the rnnt/ctc alignment
\n # https://www.youtube.com/watch?v=dgsDIuJLoJU\n #@torch.()\n def (self, encode):\n\n token_id = []\n alignment =[]\n for e in encode:\n\n id = []\n ali = []\n\n #https://github.com/DemisEom/RNNT-pytorch/blob/master/models/models.py\n #https://github.com/YooSungHyun/RNNTransducer/blob/main/networks/transducer.py\n d_id = torch.([[SOS.token_id]], ).(device=e.device)\n d, hidden = self.decoder.(d_id, None)\n\n for t in ((e)):\n ed = F.(e[t].(-) + d.(-))\n join = self.(ed)\n logit = self.(join)\n ali.(logit)\n\n i = (logit.())\n #if i == EOS.token_id: break\n\n if i != BLANK.token_id:\n id.(i)\n d_id[...] = i\n d, hidden = self.decoder.(d_id, hidden)\n\n id = torch.(id).(device=e.device)\n token_id.(id)\n #(ali)\n if (ali)>:\n alignment.(\n F.(torch.(ali),-).()\n )\n else:\n alignment.(None)\n\n #alignment = torch.(alignment)\n return token_id, alignment\n = F.softmax(ctc_logit,-)\nGreat trick Heng! I wanted to add smoothing to CTC loss but couldn't figure out how to modify the CTC source code. Adding an additional term to the loss function in addition to CTC solves the problem nicely.
", "rawMarkdown": "Great trick Heng! I wanted to add smoothing to CTC loss but couldn't figure out how to modify the CTC source code. Adding an additional term to the loss function in addition to CTC solves the problem nicely." }, { "id": 2408347, "postDate": "2023-08-25T15:27:58.667Z", "content": "Thanks for sharing very interesting techniques to enhance the score.
", "rawMarkdown": "Thanks for sharing very interesting techniques to enhance the score. " } ], "comments": [ { "id": 2407278, "author_name": "hengck23", "author_url": "", "post_date": "2023-08-25T02:21:32.563000", "content": "how prediction actually looks like:
\n
red: rnnt
\ngreen:ctc
both used same shared encoder (pretraind with ctc).. only the decoder heads changed. both decoder head had 0.795 in local CV.
\nthe red rnnt prediction is a bit blur because it uses encoder features at dim x half T (for inference sppedup).
\ngreen ctc uses dim x T. (i use T=256 frame length)
you can see EOS prediction for red rnnt (at the bottom). no PAD prediction for rnnt becuase it uses EOS termination.
\nyou can see BLANK,PAD oscillation for green CTC at the end.
the long yellow line is the BLANK prediction
\n
plot of RNNT and CTC scores
\ni estimate upper bound for ensemble : max(RNNT score, CTC score)=0.808. an improvement of 0.01. but i cannot find a way that is simple enough to fit into tflite, so i did not try and submit. (i only estimate the upper bound).
\nbasically you need to first find the non-blank time interval first (alignment), then for that alignment, average model results.
\nit is like detection ensemble, find box cluster and average
", "votes": 1, "replies": [ { "id": 2407290, "author_name": "hengck23", "author_url": "", "post_date": "2023-08-25T02:36:22.860000", "content": "code to compute the rnnt/ctc alignment
\n # https://www.youtube.com/watch?v=dgsDIuJLoJU\n #@torch.()\n def (self, encode):\n\n token_id = []\n alignment =[]\n for e in encode:\n\n id = []\n ali = []\n\n #https://github.com/DemisEom/RNNT-pytorch/blob/master/models/models.py\n #https://github.com/YooSungHyun/RNNTransducer/blob/main/networks/transducer.py\n d_id = torch.([[SOS.token_id]], ).(device=e.device)\n d, hidden = self.decoder.(d_id, None)\n\n for t in ((e)):\n ed = F.(e[t].(-) + d.(-))\n join = self.(ed)\n logit = self.(join)\n ali.(logit)\n\n i = (logit.())\n #if i == EOS.token_id: break\n\n if i != BLANK.token_id:\n id.(i)\n d_id[...] = i\n d, hidden = self.decoder.(d_id, hidden)\n\n id = torch.(id).(device=e.device)\n token_id.(id)\n #(ali)\n if (ali)>:\n alignment.(\n F.(torch.(ali),-).()\n )\n else:\n alignment.(None)\n\n #alignment = torch.(alignment)\n return token_id, alignment\n = F.softmax(ctc_logit,-)\nGreat trick Heng! I wanted to add smoothing to CTC loss but couldn't figure out how to modify the CTC source code. Adding an additional term to the loss function in addition to CTC solves the problem nicely.
", "votes": 0, "replies": [] }, { "id": 2408347, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-08-25T15:27:58.667000", "content": "Thanks for sharing very interesting techniques to enhance the score.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2407264": "i think many kagglers will reveal their trick later, but here is mine\n\nA. label smooth ====\n1. add label smooth (using kl div loss with uniform prior)\n(see this https://discuss.pytorch.org/t/label-smoothing-with-ctcloss/103392. do read the paper!)\n\nbecause label smooth limits the max probability, it forces the training to get more correct (instead of few very correct).\n\n2. slowly remove label smooth loss at the end: i.e.\n```\n total loss = a*smooth_kl_loss + (1-a)*CTC loss \n```\n\n\"a\" is scheduled from e.g. 0.01, 0.005, 0.001 and finally to be zero at the end.\n\n----\n\ni am more interested in the following two points:\n\n- how we know that long epoch training is a method\n(for me it is a bug that makes my training overnight and run longer then intended)\n\n- why? \nis it because of the blank token prediction (which is essentially the majority class)\n\n.png?generation=1692928740749828&alt=media)\n\n----\n\nwhat are your take on the last two points? thanks!\n", "2407278": "how prediction actually looks like:\n\n.png?generation=1692929804073022&alt=media)\n\n.png?generation=1692929824423460&alt=media)\n\n.png?generation=1692930446718283&alt=media)\n\n\n.png?generation=1692930844104357&alt=media)\n\n\nred: rnnt\ngreen:ctc\n\nboth used same shared encoder (pretraind with ctc).. only the decoder heads changed. both decoder head had 0.795 in local CV.\nthe red rnnt prediction is a bit blur because it uses encoder features at dim x half T (for inference sppedup).\ngreen ctc uses dim x T. (i use T=256 frame length)\n\nyou can see EOS prediction for red rnnt (at the bottom). no PAD prediction for rnnt becuase it uses EOS termination.\nyou can see BLANK,PAD oscillation for green CTC at the end.\n\nthe long yellow line is the BLANK prediction \n\n\n----\n\n\n\n.png?generation=1692929841114185&alt=media)\n\nplot of RNNT and CTC scores\n\ni estimate upper bound for ensemble : max(RNNT score, CTC score)=0.808. an improvement of 0.01. but i cannot find a way that is simple enough to fit into tflite, so i did not try and submit. (i only estimate the upper bound).\n\nbasically you need to first find the non-blank time interval first (alignment), then for that alignment, average model results.\n\nit is like detection ensemble, find box cluster and average ", "2408409": "Great trick Heng! I wanted to add smoothing to CTC loss but couldn't figure out how to modify the CTC source code. Adding an additional term to the loss function in addition to CTC solves the problem nicely.", "2408347": "Thanks for sharing very interesting techniques to enhance the score. " } }