{ "id": 434691, "title": "Anyone used competition metric as objective function?", "url": "/competitions/asl-fingerspelling/discussion/434691", "author_name": "Dieter", "post_date": "2023-08-26T06:26:12.148000", "votes": 7, "comment_count": 2, "views": 0, "content": "

Anyone made any other loss function than CrossEntropy or CTC work? We looked at MWER loss, smoothed edit distance loss but saw no benefit for our model. I worked through list of papers found under https://github.com/1ytic/edit-distance-papers

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
YearTaskReward levelAlgorithms, ModelsAffiliationAuthors, Link
2020ASRSentenceMWER, RNN-TAmazonGuo et al.
2020MTSentenceMGS, parameter searchNYUWelleck, Cho
2020ASRSentenceProper Noun, Phonetic Fuzzing, MWER, RNN-T, LASGooglePeyser, Sainath, Pundak
2019NLPSentenceGPT-2, PPO, Human labelingOpenAIZiegler, Stiennon et al.
2019ASRSentenceNeural Architecture Search, REINFORCE, CTCKPMG Nigeria, OAUBaruwa et al.
2019ASRSentenceNormalized MWERAmazonGandhe, Rastrow
2019ASRTokenMBR, RNN-TTencent, USAWeng et al.
2019ASRTokenECTC-DOCDChinaYi, Wang, Xu
2019ASRSentenceMWER, RNN-T, LASGoogleSainath, Pang et al
2019MTTokenReinforce-NAT, Non-Autoregressive TransformerChina, TencentShao, Feng et al.
2019MT, TS, APETokenLevenshtein Transformer, imitation learningFacebook, New YorkGu, Wang, Zhao
2018ASRTokenMBR, softmax margin, PAPB, S2SBrno, JHU, MERLBaskar et al.
2018ASRTokenOCD, S2SGoogle BrainSabour, Chan, Norouzi
2018ASRTokenREINFORCE, S2SNara, RIKENTjandra et al.
2018TSSentenceAlternating Actor-CriticHong Kong, TencentLi, Bing, Lam
2018ASRSentenceREINFORCE, PPO, Reward shapingTokyoPeng, Shibata, Shinozaki
2017ASRSentenceREINFORCE, Self-criticSalesforceZhou, Xiong, Socher
2017ASRSentenceMWER, LAS, Sampling, N-bestGooglePrabhavalkar et al.
2017ASRSentenceExpected Loss, RNAGoogleSak et al.
2017MTSentenceActor-Critic, Critic-awareHong Kong, New YorkGu, Cho, Li
2016ASRSentenceReward Augmented MLGoogle BrainNorouzi et al.
2016MTTokenActor-CriticMontreal, McGillBahdanau et al.
2015MTSentenceMIXERFacebookRanzato et al.
2015ASRTokenTask Loss EstimationMontreal, WrocławBahdanau et al.
2014ASRSentenceExpected Loss, CTCDeepMind, TorontoGraves, Jaitly
", "messages": [ { "id": 2409271, "postDate": "2023-08-26T06:26:12.150Z", "content": "

Anyone made any other loss function than CrossEntropy or CTC work? We looked at MWER loss, smoothed edit distance loss but saw no benefit for our model. I worked through list of papers found under https://github.com/1ytic/edit-distance-papers

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
YearTaskReward levelAlgorithms, ModelsAffiliationAuthors, Link
2020ASRSentenceMWER, RNN-TAmazonGuo et al.
2020MTSentenceMGS, parameter searchNYUWelleck, Cho
2020ASRSentenceProper Noun, Phonetic Fuzzing, MWER, RNN-T, LASGooglePeyser, Sainath, Pundak
2019NLPSentenceGPT-2, PPO, Human labelingOpenAIZiegler, Stiennon et al.
2019ASRSentenceNeural Architecture Search, REINFORCE, CTCKPMG Nigeria, OAUBaruwa et al.
2019ASRSentenceNormalized MWERAmazonGandhe, Rastrow
2019ASRTokenMBR, RNN-TTencent, USAWeng et al.
2019ASRTokenECTC-DOCDChinaYi, Wang, Xu
2019ASRSentenceMWER, RNN-T, LASGoogleSainath, Pang et al
2019MTTokenReinforce-NAT, Non-Autoregressive TransformerChina, TencentShao, Feng et al.
2019MT, TS, APETokenLevenshtein Transformer, imitation learningFacebook, New YorkGu, Wang, Zhao
2018ASRTokenMBR, softmax margin, PAPB, S2SBrno, JHU, MERLBaskar et al.
2018ASRTokenOCD, S2SGoogle BrainSabour, Chan, Norouzi
2018ASRTokenREINFORCE, S2SNara, RIKENTjandra et al.
2018TSSentenceAlternating Actor-CriticHong Kong, TencentLi, Bing, Lam
2018ASRSentenceREINFORCE, PPO, Reward shapingTokyoPeng, Shibata, Shinozaki
2017ASRSentenceREINFORCE, Self-criticSalesforceZhou, Xiong, Socher
2017ASRSentenceMWER, LAS, Sampling, N-bestGooglePrabhavalkar et al.
2017ASRSentenceExpected Loss, RNAGoogleSak et al.
2017MTSentenceActor-Critic, Critic-awareHong Kong, New YorkGu, Cho, Li
2016ASRSentenceReward Augmented MLGoogle BrainNorouzi et al.
2016MTTokenActor-CriticMontreal, McGillBahdanau et al.
2015MTSentenceMIXERFacebookRanzato et al.
2015ASRTokenTask Loss EstimationMontreal, WrocławBahdanau et al.
2014ASRSentenceExpected Loss, CTCDeepMind, TorontoGraves, Jaitly
", "rawMarkdown": "Anyone made any other loss function than CrossEntropy or CTC work? We looked at MWER loss, smoothed edit distance loss but saw no benefit for our model. I worked through list of papers found under https://github.com/1ytic/edit-distance-papers\n\n\n\n| Year | Task | Reward level | Algorithms, Models | Affiliation | Authors, Link |\n|:----:|:----:|--------------|--------------------|-------------|---------------|\n| 2020 | ASR | Sentence | MWER, RNN-T | Amazon | [Guo et al.](https://arxiv.org/abs/2007.13802)|\n| 2020 | MT | Sentence | MGS, parameter search | NYU | [Welleck, Cho](https://arxiv.org/abs/2006.03158)|\n| 2020 | ASR | Sentence | Proper Noun, Phonetic Fuzzing, MWER, RNN-T, LAS | Google | [Peyser, Sainath, Pundak](https://arxiv.org/abs/2005.09756)|\n| 2019 | NLP | Sentence | GPT-2, PPO, Human labeling | OpenAI | [Ziegler, Stiennon et al.](https://arxiv.org/abs/1909.08593)|\n| 2019 | ASR | Sentence | Neural Architecture Search, REINFORCE, CTC | KPMG Nigeria, OAU | [Baruwa et al.](https://arxiv.org/abs/1912.05946)|\n| 2019 | ASR | Sentence | Normalized MWER | Amazon | [Gandhe, Rastrow](https://arxiv.org/abs/1912.03363)|\n| 2019 | ASR | Token | MBR, RNN-T | Tencent, USA | [Weng et al.](https://arxiv.org/abs/1911.12487)|\n| 2019 | ASR | Token | ECTC-DOCD | China | [Yi, Wang, Xu](https://www.isca-speech.org/archive/Interspeech_2019/pdfs/1212.pdf)|\n| 2019 | ASR | Sentence | MWER, RNN-T, LAS | Google | [Sainath, Pang et al](https://arxiv.org/abs/1908.10992)|\n| 2019 | MT | Token | Reinforce-NAT, Non-Autoregressive Transformer | China, Tencent | [Shao, Feng et al.](https://arxiv.org/abs/1906.09444)|\n| 2019 | MT, TS, APE | Token | Levenshtein Transformer, imitation learning | Facebook, New York | [Gu, Wang, Zhao](https://arxiv.org/abs/1905.11006)|\n| 2018 | ASR | Token | MBR, softmax margin, PAPB, S2S | Brno, JHU, MERL | [Baskar et al.](https://arxiv.org/abs/1811.02770)|\n| 2018 | ASR | Token | OCD, S2S | Google Brain | [Sabour, Chan, Norouzi](https://arxiv.org/abs/1810.01398)|\n| 2018 | ASR | Token | REINFORCE, S2S | Nara, RIKEN | [Tjandra et al.](https://ahcweb01.naist.jp/papers/journal/2019/201906_IEEE_andros-tj_1/201906_IEEE_andros-tj_1.paper.pdf)|\n| 2018 | TS | Sentence | Alternating Actor-Critic | Hong Kong, Tencent | [Li, Bing, Lam](https://arxiv.org/abs/1803.11070)|\n| 2018 | ASR | Sentence | REINFORCE, PPO, Reward shaping | Tokyo |[Peng, Shibata, Shinozaki](http://www.apsipa.org/proceedings/2018/pdfs/0001934.pdf)|\n| 2017 | ASR | Sentence | REINFORCE, Self-critic | Salesforce | [Zhou, Xiong, Socher](https://arxiv.org/abs/1712.07101)|\n| 2017 | ASR | Sentence | MWER, LAS, Sampling, N-best | Google |[Prabhavalkar et al.](https://arxiv.org/abs/1712.01818)|\n| 2017 | ASR | Sentence | Expected Loss, RNA | Google | [Sak et al.](https://pdfs.semanticscholar.org/7703/a2c5468ecbee5b62c048339a03358ed5fe19.pdf)|\n| 2017 | MT | Sentence | Actor-Critic, Critic-aware | Hong Kong, New York | [Gu, Cho, Li](https://arxiv.org/abs/1702.02429)|\n| 2016 | ASR | Sentence | Reward Augmented ML | Google Brain | [Norouzi et al.](https://arxiv.org/abs/1609.00150)|\n| 2016 | MT | Token | Actor-Critic | Montreal, McGill | [Bahdanau et al.](https://arxiv.org/abs/1607.07086)|\n| 2015 | MT | Sentence | MIXER | Facebook | [Ranzato et al.](https://arxiv.org/abs/1511.06732)|\n| 2015 | ASR | Token | Task Loss Estimation | Montreal, Wrocław | [Bahdanau et al.](https://arxiv.org/abs/1511.06456)|\n| 2014 | ASR | Sentence | Expected Loss, CTC | DeepMind, Toronto | [Graves, Jaitly](http://proceedings.mlr.press/v32/graves14.pdf)|\n", "votes": 7 }, { "id": 2411231, "postDate": "2023-08-27T13:41:37.953Z", "content": "

Just posted my solution - I got MWER training to work but the contribution on the final ensemble was modest: +0.001 - +0.002. This was with beam search with a size 5 beam. With greedy decoding gains were around +0.005 in local eval. When I used weaker (underfitted?) models, the gains were fairly sizeable. I also used it as an auxiliary loss, but it had no effect except for slowing things down massively 😀
\nI pretty much did it as in Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models. The normalization was important as was using a small lambda (0.01).
\nBut the whole procedure wasn't very stable, even after optimizing params, results varried a lot and the best result was usually achieved within 4 epochs with a small LR around 1e-5.

\n

Congratz on your win!

", "rawMarkdown": "Just posted my solution - I got MWER training to work but the contribution on the final ensemble was modest: +0.001 - +0.002. This was with beam search with a size 5 beam. With greedy decoding gains were around +0.005 in local eval. When I used weaker (underfitted?) models, the gains were fairly sizeable. I also used it as an auxiliary loss, but it had no effect except for slowing things down massively 😀\nI pretty much did it as in [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://arxiv.org/pdf/1712.01818.pdf). The normalization was important as was using a small lambda (0.01).\nBut the whole procedure wasn't very stable, even after optimizing params, results varried a lot and the best result was usually achieved within 4 epochs with a small LR around 1e-5.\n\nCongratz on your win!", "votes": 1, "replies": [ { "id": 2411790, "postDate": "2023-08-27T21:25:16.843Z", "content": "

Interesting. I also got MWER training to work technically. But general stability is an issue. I just tried it as a combi loss with normal CrossEntropy already when start training from scratch, but might have been better to finetune some good models with it for a few epochs. When using batched edit distance it was quite fast. I took it from this inconspicuous repository: https://github.com/francescocastelli/torchdistance
\nI was also wondering if not the time shifts introduced with naïve cutmix (i.e. not correctly aligned cuts of sequence and phrases) already make CrossEntropy more robust in terms of time-shifts, and thats the reason we did not see benefits from MWER training.

", "rawMarkdown": "Interesting. I also got MWER training to work technically. But general stability is an issue. I just tried it as a combi loss with normal CrossEntropy already when start training from scratch, but might have been better to finetune some good models with it for a few epochs. When using batched edit distance it was quite fast. I took it from this inconspicuous repository: https://github.com/francescocastelli/torchdistance\nI was also wondering if not the time shifts introduced with naïve cutmix (i.e. not correctly aligned cuts of sequence and phrases) already make CrossEntropy more robust in terms of time-shifts, and thats the reason we did not see benefits from MWER training. ", "votes": 1 } ] } ], "comments": [ { "id": 2411231, "author_name": "flg", "author_url": "", "post_date": "2023-08-27T13:41:37.953000", "content": "

Just posted my solution - I got MWER training to work but the contribution on the final ensemble was modest: +0.001 - +0.002. This was with beam search with a size 5 beam. With greedy decoding gains were around +0.005 in local eval. When I used weaker (underfitted?) models, the gains were fairly sizeable. I also used it as an auxiliary loss, but it had no effect except for slowing things down massively 😀
\nI pretty much did it as in Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models. The normalization was important as was using a small lambda (0.01).
\nBut the whole procedure wasn't very stable, even after optimizing params, results varried a lot and the best result was usually achieved within 4 epochs with a small LR around 1e-5.

\n

Congratz on your win!

", "votes": 1, "replies": [ { "id": 2411790, "author_name": "Dieter", "author_url": "", "post_date": "2023-08-27T21:25:16.843000", "content": "

Interesting. I also got MWER training to work technically. But general stability is an issue. I just tried it as a combi loss with normal CrossEntropy already when start training from scratch, but might have been better to finetune some good models with it for a few epochs. When using batched edit distance it was quite fast. I took it from this inconspicuous repository: https://github.com/francescocastelli/torchdistance
\nI was also wondering if not the time shifts introduced with naïve cutmix (i.e. not correctly aligned cuts of sequence and phrases) already make CrossEntropy more robust in terms of time-shifts, and thats the reason we did not see benefits from MWER training.

", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2409271": "Anyone made any other loss function than CrossEntropy or CTC work? We looked at MWER loss, smoothed edit distance loss but saw no benefit for our model. I worked through list of papers found under https://github.com/1ytic/edit-distance-papers\n\n\n\n| Year | Task | Reward level | Algorithms, Models | Affiliation | Authors, Link |\n|:----:|:----:|--------------|--------------------|-------------|---------------|\n| 2020 | ASR | Sentence | MWER, RNN-T | Amazon | [Guo et al.](https://arxiv.org/abs/2007.13802)|\n| 2020 | MT | Sentence | MGS, parameter search | NYU | [Welleck, Cho](https://arxiv.org/abs/2006.03158)|\n| 2020 | ASR | Sentence | Proper Noun, Phonetic Fuzzing, MWER, RNN-T, LAS | Google | [Peyser, Sainath, Pundak](https://arxiv.org/abs/2005.09756)|\n| 2019 | NLP | Sentence | GPT-2, PPO, Human labeling | OpenAI | [Ziegler, Stiennon et al.](https://arxiv.org/abs/1909.08593)|\n| 2019 | ASR | Sentence | Neural Architecture Search, REINFORCE, CTC | KPMG Nigeria, OAU | [Baruwa et al.](https://arxiv.org/abs/1912.05946)|\n| 2019 | ASR | Sentence | Normalized MWER | Amazon | [Gandhe, Rastrow](https://arxiv.org/abs/1912.03363)|\n| 2019 | ASR | Token | MBR, RNN-T | Tencent, USA | [Weng et al.](https://arxiv.org/abs/1911.12487)|\n| 2019 | ASR | Token | ECTC-DOCD | China | [Yi, Wang, Xu](https://www.isca-speech.org/archive/Interspeech_2019/pdfs/1212.pdf)|\n| 2019 | ASR | Sentence | MWER, RNN-T, LAS | Google | [Sainath, Pang et al](https://arxiv.org/abs/1908.10992)|\n| 2019 | MT | Token | Reinforce-NAT, Non-Autoregressive Transformer | China, Tencent | [Shao, Feng et al.](https://arxiv.org/abs/1906.09444)|\n| 2019 | MT, TS, APE | Token | Levenshtein Transformer, imitation learning | Facebook, New York | [Gu, Wang, Zhao](https://arxiv.org/abs/1905.11006)|\n| 2018 | ASR | Token | MBR, softmax margin, PAPB, S2S | Brno, JHU, MERL | [Baskar et al.](https://arxiv.org/abs/1811.02770)|\n| 2018 | ASR | Token | OCD, S2S | Google Brain | [Sabour, Chan, Norouzi](https://arxiv.org/abs/1810.01398)|\n| 2018 | ASR | Token | REINFORCE, S2S | Nara, RIKEN | [Tjandra et al.](https://ahcweb01.naist.jp/papers/journal/2019/201906_IEEE_andros-tj_1/201906_IEEE_andros-tj_1.paper.pdf)|\n| 2018 | TS | Sentence | Alternating Actor-Critic | Hong Kong, Tencent | [Li, Bing, Lam](https://arxiv.org/abs/1803.11070)|\n| 2018 | ASR | Sentence | REINFORCE, PPO, Reward shaping | Tokyo |[Peng, Shibata, Shinozaki](http://www.apsipa.org/proceedings/2018/pdfs/0001934.pdf)|\n| 2017 | ASR | Sentence | REINFORCE, Self-critic | Salesforce | [Zhou, Xiong, Socher](https://arxiv.org/abs/1712.07101)|\n| 2017 | ASR | Sentence | MWER, LAS, Sampling, N-best | Google |[Prabhavalkar et al.](https://arxiv.org/abs/1712.01818)|\n| 2017 | ASR | Sentence | Expected Loss, RNA | Google | [Sak et al.](https://pdfs.semanticscholar.org/7703/a2c5468ecbee5b62c048339a03358ed5fe19.pdf)|\n| 2017 | MT | Sentence | Actor-Critic, Critic-aware | Hong Kong, New York | [Gu, Cho, Li](https://arxiv.org/abs/1702.02429)|\n| 2016 | ASR | Sentence | Reward Augmented ML | Google Brain | [Norouzi et al.](https://arxiv.org/abs/1609.00150)|\n| 2016 | MT | Token | Actor-Critic | Montreal, McGill | [Bahdanau et al.](https://arxiv.org/abs/1607.07086)|\n| 2015 | MT | Sentence | MIXER | Facebook | [Ranzato et al.](https://arxiv.org/abs/1511.06732)|\n| 2015 | ASR | Token | Task Loss Estimation | Montreal, Wrocław | [Bahdanau et al.](https://arxiv.org/abs/1511.06456)|\n| 2014 | ASR | Sentence | Expected Loss, CTC | DeepMind, Toronto | [Graves, Jaitly](http://proceedings.mlr.press/v32/graves14.pdf)|\n", "2411231": "Just posted my solution - I got MWER training to work but the contribution on the final ensemble was modest: +0.001 - +0.002. This was with beam search with a size 5 beam. With greedy decoding gains were around +0.005 in local eval. When I used weaker (underfitted?) models, the gains were fairly sizeable. I also used it as an auxiliary loss, but it had no effect except for slowing things down massively 😀\nI pretty much did it as in [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://arxiv.org/pdf/1712.01818.pdf). The normalization was important as was using a small lambda (0.01).\nBut the whole procedure wasn't very stable, even after optimizing params, results varried a lot and the best result was usually achieved within 4 epochs with a small LR around 1e-5.\n\nCongratz on your win!" } }