My solution is quite simple. I wasn't going to write about it, but there are no Jasper-like solutions in discussions yet :)
\nPreprocessing was almost the same with previous competition. But deleted mixup and replace augmentation. And add more complex time augmentations.
\nI only used CTC loss.
\nRandom split not by user for 20 folds grouping by data length.
\nDeleted obviously bad samples:
supplemental_metadata_df = supplemental_metadata_df[~((supplemental_metadata_df[] < ) &\n (supplemental_metadata_df[] > ))]\n\ntrain_df = train_df[~((train_df[] < ) & (train_df[] > ))]\n
\nUsed all supplemental data in train.
Also tried to create second loader with pseudo labels on bad predicted samples. Or delete bad predicted samples. Used both loaders while training:
\nloader_to_use = \n epoch % == :\n loader_to_use = \nBut it didn't help much.
\nFirst I tried simple quartznet, but with all strides set to 1.
\nThen I added some features step by step and trained each experiment from previous best checkpoint.
When I used raw conformer and it has very low score. But when I changed input 2D conv to 1D conv or when I just deleted it - it was much better, but still little worse than my Jasper models.
\nThanks to Kaggle for this great competition. Thanks to all top participants who shared their awesome solutions.
\nCongratulations to all winners!
My solution is quite simple. I wasn't going to write about it, but there are no Jasper-like solutions in discussions yet :)
\nPreprocessing was almost the same with previous competition. But deleted mixup and replace augmentation. And add more complex time augmentations.
\nI only used CTC loss.
\nRandom split not by user for 20 folds grouping by data length.
\nDeleted obviously bad samples:
supplemental_metadata_df = supplemental_metadata_df[~((supplemental_metadata_df[] < ) &\n (supplemental_metadata_df[] > ))]\n\ntrain_df = train_df[~((train_df[] < ) & (train_df[] > ))]\n
\nUsed all supplemental data in train.
Also tried to create second loader with pseudo labels on bad predicted samples. Or delete bad predicted samples. Used both loaders while training:
\nloader_to_use = \n epoch % == :\n loader_to_use = \nBut it didn't help much.
\nFirst I tried simple quartznet, but with all strides set to 1.
\nThen I added some features step by step and trained each experiment from previous best checkpoint.
When I used raw conformer and it has very low score. But when I changed input 2D conv to 1D conv or when I just deleted it - it was much better, but still little worse than my Jasper models.
\nThanks to Kaggle for this great competition. Thanks to all top participants who shared their awesome solutions.
\nCongratulations to all winners!
Great work @kolyaforrat did you develop in pytorch and port to tf ? I understand nemo jasper models are only in pytorch.
", "rawMarkdown": "Great work @kolyaforrat did you develop in pytorch and port to tf ? I understand nemo jasper models are only in pytorch. ", "replies": [ { "id": 2417778, "postDate": "2023-08-31T20:36:42.260Z", "content": "Yes, I developed pytorch, and converted it with nobuco. But didn't use Nemo, wrote my own copying some blocks and logic from Nemo.
\nOne of the reasons to write my own - is ability to convert this models with nobuco without a lot of changes
Congratulations. Thanks for sharing very useful information regarding your solution and your experience from the competition.
\n\"train more epochs\"- This is limited by the notebook time of <9hrs. I once trained my model over many epochs and I could not submit the same notebook due to time restrictions. I think clever part is to separate training and inference notebooks.
\nHow to enforce fixed length of input?
\n\nHow to enforce fixed length of input?
\n
TORCH.NN.FUNCTIONAL.INTERPOLATE
", "rawMarkdown": ">How to enforce fixed length of input?\n\n[TORCH.NN.FUNCTIONAL.INTERPOLATE](https://pytorch.org/docs/stable/generated/torch.nn.functional.interpolate.html)" } ] } ], "comments": [ { "id": 2417184, "author_name": "Darragh", "author_url": "", "post_date": "2023-08-31T13:22:27.897000", "content": "Great work @kolyaforrat did you develop in pytorch and port to tf ? I understand nemo jasper models are only in pytorch.
", "votes": 0, "replies": [ { "id": 2417778, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-08-31T20:36:42.260000", "content": "Yes, I developed pytorch, and converted it with nobuco. But didn't use Nemo, wrote my own copying some blocks and logic from Nemo.
\nOne of the reasons to write my own - is ability to convert this models with nobuco without a lot of changes
Congratulations. Thanks for sharing very useful information regarding your solution and your experience from the competition.
\n\"train more epochs\"- This is limited by the notebook time of <9hrs. I once trained my model over many epochs and I could not submit the same notebook due to time restrictions. I think clever part is to separate training and inference notebooks.
\nHow to enforce fixed length of input?
\n\n", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2409962": "My solution is quite simple. I wasn't going to write about it, but there are no Jasper-like solutions in discussions yet :)\n\n# Preprocessing \nPreprocessing was almost the same with [previous competition](https://www.kaggle.com/competitions/asl-signs/discussion/406306). But deleted mixup and replace augmentation. And add more complex time augmentations.\n\n# Loss\nI only used CTC loss.\n\n# Data split\nRandom split **not** by user for 20 folds grouping by data length.\nDeleted obviously bad samples:\n```python\nsupplemental_metadata_df = supplemental_metadata_df[~((supplemental_metadata_df['length'] < 10) &\n (supplemental_metadata_df['phrase_len'] > 8))]\n\ntrain_df = train_df[~((train_df['length'] < 7) & (train_df['phrase_len'] > 5))]\n``` \nUsed all supplemental data in train.\n\nAlso tried to create second loader with pseudo labels on bad predicted samples. Or delete bad predicted samples. Used both loaders while training:\n\n```python\nloader_to_use = 'main_loader'\nif epoch % 3 == 0:\n loader_to_use = 'second_loader'\n```\nBut it didn't help much.\n\n# Model\nFirst I tried simple quartznet, but with all strides set to 1.\nThen I added some features step by step and trained each experiment from previous best checkpoint.\n\n### Same for all runs:\n* Optimizer: LookAheadAdamW\n* Scheduler: Onecycle\n* Batch size: 80\n\n### List of experiments:\n* Simple 5x5 quartznet with stride = 1. [lr = 8e-3] [epochs=200] CV:0.8035\n* Simple 5x5 quartznet with stride = 1. [lr = 7e-3] [epochs=150] CV: 0.8106\n* Previous + SE blocks. [lr = 6e-3] [epochs=171] CV: 0.8160\n* Previous + Deep supervision (DSV) outputs for train loss. [lr = 5.8e-3] [epochs=175] CV: 0.8211\n* Previous without DSV + higher aug probabilities. Note: score worse but it needed to train one checkpoint without DSV for better next train results. [lr = 6e-3] [epochs=180] CV: 0.8177\n* Previous + DSV + increased dropout. [lr = 6.1e-3] [epochs=200] CV: 0.8237\n* Previous - DSV + lower dropout. [lr = 6e-3] [epochs=173] CV: 0.8188\n* Previous + Hyper Column (HYP) + 2 more blocks with 5 repeats. [lr=5.8e-3] [epochs=180] CV: 0.8252\n* Previous + masked 1conv + masked SE + 2D DepthWise CNN layer before classifier + more dropout. [lr=5.8e-3] [epochs=200] CV: 0.8294\n* Previous + lstm after 2D Conv. [lr=1.75e-3] [epochs=125] CV: 0.8301\n\n\n# What didn't work for me:\n* Conformer models. I tried it but I guess without enough effort.\n* Adding attention or conformer blocks to Jasper models.\n\nWhen I used raw conformer and it has very low score. But when I changed input 2D conv to 1D conv or when I just deleted it - it was much better, but still little worse than my Jasper models.\n\n# What I should have done, but didn't:\n* Train more epochs\n* Try better with conformer or squeezeformer\n* Add post processing\n* Try seq2seq\n* Fixed length of input\n\nThanks to Kaggle for this great competition. Thanks to all top participants who shared their awesome solutions. \nCongratulations to all winners! \n", "2417184": "Great work @kolyaforrat did you develop in pytorch and port to tf ? I understand nemo jasper models are only in pytorch. ", "2412129": "Congratulations. Thanks for sharing very useful information regarding your solution and your experience from the competition.\n\"train more epochs\"- This is limited by the notebook time of <9hrs. I once trained my model over many epochs and I could not submit the same notebook due to time restrictions. I think clever part is to separate training and inference notebooks.\nHow to enforce fixed length of input?" } }How to enforce fixed length of input?
\n