Fixed-length input (220 frames), padding shorter inputs and resizing longer inputs.
\n2 layer MLP landmark encoder + 6 layer 384-dim Conformer + 1 layer GRU and 500 epochs training (takes ~8 hours on Kaggle TPUs).
\nPost-processing (+0.003 in CV and LB) by:
len() <= :\n = + \nMy notebook is available at https://www.kaggle.com/code/nightsh4de/ctc-transformer/notebook
\nIt is primarily based on the previous 1st solution https://www.kaggle.com/code/hoyso48/1st-place-solution-training and some public notebooks of this competition:
\nhttps://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference#Landmark-Embedding
\nhttps://www.kaggle.com/code/irohith/aslfr-transformer
\nhttps://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu. Many many thanks to them.
Basically the same as the previous 1st solution. But I found using 3d positions (means including depth) and pose landmarks gives better CV and LB score.
\nMy input: left-right hand, eye, nose, lips and pose landmarks.
I believe this task would be quite similar to Automatic Speech Recognition (ASR), so I used Conformer https://arxiv.org/abs/2005.08100
\nFixed-length input (220 frames), padding shorter inputs and resizing longer inputs.
\n2 layer MLP landmark encoder + 6 layer 384-dim Conformer + 1 layer GRU and 500 epochs training (takes ~8 hours on Kaggle TPUs).
\nPost-processing (+0.003 in CV and LB) by:
len() <= :\n = + \nMy notebook is available at https://www.kaggle.com/code/nightsh4de/ctc-transformer/notebook
\nIt is primarily based on the previous 1st solution https://www.kaggle.com/code/hoyso48/1st-place-solution-training and some public notebooks of this competition:
\nhttps://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference#Landmark-Embedding
\nhttps://www.kaggle.com/code/irohith/aslfr-transformer
\nhttps://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu. Many many thanks to them.
Basically the same as the previous 1st solution. But I found using 3d positions (means including depth) and pose landmarks gives better CV and LB score.
\nMy input: left-right hand, eye, nose, lips and pose landmarks.
I believe this task would be quite similar to Automatic Speech Recognition (ASR), so I used Conformer https://arxiv.org/abs/2005.08100
\nCongratulations. Thanks for sharing very interesting details.
", "rawMarkdown": "Congratulations. Thanks for sharing very interesting details.", "votes": 1 }, { "id": 2407250, "postDate": "2023-08-25T01:40:56.083Z", "content": "\"Masking for variable-length input …\"
\nactually you can do without masking. for example, whisper ASR uses fixed size 30 sec audio segments. it pads if it is too short.
\ni find this kind of framework simplifies model and training. but there is one parameter that can be learned: the pad value (more correctly, pad embedding)
But masking is faster when processing shorter inputs while being able to take longer inputs without resizing. Am I right?
", "rawMarkdown": "But masking is faster when processing shorter inputs while being able to take longer inputs without resizing. Am I right?" } ] } ], "comments": [ { "id": 2408356, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-08-25T15:31:39.840000", "content": "Congratulations. Thanks for sharing very interesting details.
", "votes": 1, "replies": [] }, { "id": 2407250, "author_name": "hengck23", "author_url": "", "post_date": "2023-08-25T01:40:56.083000", "content": "\"Masking for variable-length input …\"
\nactually you can do without masking. for example, whisper ASR uses fixed size 30 sec audio segments. it pads if it is too short.
\ni find this kind of framework simplifies model and training. but there is one parameter that can be learned: the pad value (more correctly, pad embedding)
But masking is faster when processing shorter inputs while being able to take longer inputs without resizing. Am I right?
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2407228": "\n# TL; DR\n\nFixed-length input (220 frames), padding shorter inputs and resizing longer inputs.\n\n2 layer MLP landmark encoder + 6 layer 384-dim Conformer + 1 layer GRU and 500 epochs training (takes ~8 hours on Kaggle TPUs).\n\nPost-processing (+0.003 in CV and LB) by:\n```\nif len(pred) <= 4:\n pred = pred + \" -aero\"\n```\n\nMy notebook is available at https://www.kaggle.com/code/nightsh4de/ctc-transformer/notebook\nIt is primarily based on the previous 1st solution https://www.kaggle.com/code/hoyso48/1st-place-solution-training and some public notebooks of this competition:\n https://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference#Landmark-Embedding\nhttps://www.kaggle.com/code/irohith/aslfr-transformer \nhttps://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu. Many many thanks to them.\n\n# Data Preprocessing and Augmentation\n\nBasically the same as the previous 1st solution. But I found using 3d positions (means including depth) and pose landmarks gives better CV and LB score.\n\nMy input: left-right hand, eye, nose, lips and pose landmarks.\n\n# Model\n\nI believe this task would be quite similar to Automatic Speech Recognition (ASR), so I used Conformer https://arxiv.org/abs/2005.08100\n\n# Post-preprocessing\n\n1. I checked my worst predictions in the validation set and found that shorter predictions are worse. And most very short predictions (length less than 5) are basically predicting nothing, e.g. single characters like \"a\" or space \" \".\n2. I found that only few label's length is less or equal to 5. \n3. So adding some make-up phrases to very short predictions might be a good idea.\n3. I picked the most common chars in training set: \"a\", \"e\", \"r\", \"o\", \"-\", \" \".\n4. I test all the combinations of \"a\", \"e\", \"r\", \"o\", \"-\", \" \" based on validation set and \" -aero\" is the best.\n\n# What not worked\n\n1. Autoregressive transformer models. \n\nI spent most of my time on transformers. But they strongly overfits the phrase and I couldn't find a way to solve it.\n\nThe problem is , for an input X[1...n] and its phrase \"123456789\". We may expect our model predict \"12345\" for the first half input X[1...n/2]. \n\nFor CTC models, it is true. But my autoregressive model always makes half correct prediction \"12345\" + half random incorrect prediction e.g. some random 5-digit numbers.\n\n2. Masking for variable-length input\n\nI tried to use the same masking techniques as the previous 1st solution https://www.kaggle.com/code/hoyso48/1st-place-solution-training, to support longer input frames.\n\nHowever, masking models always produce lower CV and LB scores (-0.02). I believe it is due to the masking limits the output length, but shorter inputs still require a longer output space. Although, unfortunately, I don't have time to verify it, I believe it might enable much better solutions.\n\n3. Empty embedding for fixed-length input\n\nI tried to use learnable constant weights for padded empty frames and 2-layer MLP for encoding landmarks in original frames. It seems to me makes more sense, but CV score is worse than direct encoding all frames. \n\n\n", "2408356": "Congratulations. Thanks for sharing very interesting details.", "2407250": "\"Masking for variable-length input ...\"\nactually you can do without masking. for example, whisper ASR uses fixed size 30 sec audio segments. it pads if it is too short.\ni find this kind of framework simplifies model and training. but there is one parameter that can be learned: the pad value (more correctly, pad embedding)" } }