Firstly, I'd like to extend my deepest gratitude to my teammates:
\nTheir enormous effort has been instrumental in this competition.
\nI'd like also thanks to the authors of following notebooks. Our success wouldn't have been possible without the insights and techniques we adopted from these notebooks. A big thanks to you.
\n@irohith for his excellent preprocessing and training notebooks:
\n@hoyso48 for his 1st place solution from a previous competition:
\n@greysnow for the CTC on TPU notebook:
\n@royalacecat for the notebook on more blocks with quantization:
\nOur solution is based on the CTC open source notebook. Here are the key improvements we introduced (all scores are public LB):
\nReferring to @irohith's preprocessing notebook, we made changes by adding more face and pose indices. We selected the new indices based on their variance — higher variance indicates more information.
\nInitially, the indices were:
\nLIP = [\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n]\nLPOSE = [, , , , ]\nRPOSE = [, , , , ]\nPOSE = LPOSE + RPOSE\nWe then updated them to:
\nLIP = [\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n]\n\n\nface_id = [,,,,,,,,,,,\n ,,,,,,,,,,,,\n ,,,,,,,,,]\n\n\n k face_id:\n k LIP:\n LIP.append(k)\n\n\nl = (LIP)\nLIP = LIP[:(l - l/)]\n\n\nLPOSE = [, , , , , , , , , , , , , , , ]\nRPOSE = [, , , , , , , , , , , , , , , ]\nPOSE = LPOSE + RPOSE\nfilter out full Nan frame
\norigin notebook:
\nhand = tf.concat([rhand, lhand], axis=)\nhand = tf.where(tf.math.is_nan(hand), , hand)\nmask = tf.math.not_equal(tf.reduce_sum(hand, axis=[, ]), )\nupdate to
\nhand = tf.concat([rhand, lhand,lip,rpose,lpose], axis=)\nhand = tf.where(tf.math.is_nan(hand), , hand)\nmask = tf.math.not_equal(tf.reduce_sum(hand, axis=[, ]), )\nThose new indices along with 256 frame length gives us about 0.02 LB improvement!
\nsee 1st Place Solution Training
\nTo enhance our model's stability during training, we implemented a CTC loss with an added regularization component. By combining the CTC loss with a Kullback-Leibler (KL) divergence, we introduced label smoothing to the model. Smooth weight = 0.7 works best for our model
\n ():\n \n label_length = tf.reduce_sum(tf.cast(labels != pad_token_idx, tf.int32), axis=-)\n logit_length = tf.ones(tf.shape(logits)[], dtype=tf.int32) * tf.shape(logits)[]\n ctc_loss = tf.nn.ctc_loss(\n labels=labels,\n logits=logits,\n label_length=label_length,\n logit_length=logit_length,\n blank_index=pad_token_idx,\n logits_time_major=\n )\n ctc_loss = tf.reduce_mean(ctc_loss)\n\n \n kl_inp = tf.nn.softmax(logits)\n\n \n kl_tar = tf.fill(tf.shape(logits), / num_classes)\n\n \n kldiv_loss = (tf.keras.losses.KLDivergence(tf.keras.losses.Reduction.NONE)(kl_tar, kl_inp) \n + tf.keras.losses.KLDivergence(tf.keras.losses.Reduction.NONE)(kl_inp, kl_tar))/\n\n kldiv_loss = tf.reduce_mean(kldiv_loss)\n\n \n loss = ( - weight) * ctc_loss + weight * kldiv_loss\n loss\nDuring our experimentation, we found a significant insight: training solely on the supplemental dataset yielded a LB score of 0.367. This highlighted the potential value embedded within the supplemental data.
\nWe decided to first pre-train our model on the supplemental dataset, then loaded these pre-trained weights to further train the model on the main dataset. We tested multiple epochs: 40, 60, and 80 for pre-train. 80 epochs works best and give a improvement of 0.013 on the LB score.
\nAugmentation:
\nIn 2nd place solution, @hoyso48 mention that some augmentation didn't improve the model during short epoch training but in longer training would have the effect. This could applied to our case as well but we haven't verified yet.
Switching to GIC-CTC:
\nhttps://www.kaggle.com/code/wuhongrui/aslfr-gcs-path
\nhttps://www.kaggle.com/wuhongrui/aslfr-training
\nhttps://www.kaggle.com/wuhongrui/ctc-final-inference-notebook
Training notebook is written specific for training on Google Colab. Frist use gcs-path notebook check the path in training notebook. Set USE_SUPPLY = True first to train supply set, then set USE_VAL = True to load pre-train weight and train main data set.
\nTo use it on Kaggle TPU, change the file address to corresponding Kaggle input address and you NEED to modify the CTC loss. Check ASLFR CTC on TPU
", "messages": [ { "id": 2410560, "postDate": "2023-08-27T04:50:46.080Z", "content": "Firstly, I'd like to extend my deepest gratitude to my teammates:
\nTheir enormous effort has been instrumental in this competition.
\nI'd like also thanks to the authors of following notebooks. Our success wouldn't have been possible without the insights and techniques we adopted from these notebooks. A big thanks to you.
\n@irohith for his excellent preprocessing and training notebooks:
\n@hoyso48 for his 1st place solution from a previous competition:
\n@greysnow for the CTC on TPU notebook:
\n@royalacecat for the notebook on more blocks with quantization:
\nOur solution is based on the CTC open source notebook. Here are the key improvements we introduced (all scores are public LB):
\nReferring to @irohith's preprocessing notebook, we made changes by adding more face and pose indices. We selected the new indices based on their variance — higher variance indicates more information.
\nInitially, the indices were:
\nLIP = [\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n]\nLPOSE = [, , , , ]\nRPOSE = [, , , , ]\nPOSE = LPOSE + RPOSE\nWe then updated them to:
\nLIP = [\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n]\n\n\nface_id = [,,,,,,,,,,,\n ,,,,,,,,,,,,\n ,,,,,,,,,]\n\n\n k face_id:\n k LIP:\n LIP.append(k)\n\n\nl = (LIP)\nLIP = LIP[:(l - l/)]\n\n\nLPOSE = [, , , , , , , , , , , , , , , ]\nRPOSE = [, , , , , , , , , , , , , , , ]\nPOSE = LPOSE + RPOSE\nfilter out full Nan frame
\norigin notebook:
\nhand = tf.concat([rhand, lhand], axis=)\nhand = tf.where(tf.math.is_nan(hand), , hand)\nmask = tf.math.not_equal(tf.reduce_sum(hand, axis=[, ]), )\nupdate to
\nhand = tf.concat([rhand, lhand,lip,rpose,lpose], axis=)\nhand = tf.where(tf.math.is_nan(hand), , hand)\nmask = tf.math.not_equal(tf.reduce_sum(hand, axis=[, ]), )\nThose new indices along with 256 frame length gives us about 0.02 LB improvement!
\nsee 1st Place Solution Training
\nTo enhance our model's stability during training, we implemented a CTC loss with an added regularization component. By combining the CTC loss with a Kullback-Leibler (KL) divergence, we introduced label smoothing to the model. Smooth weight = 0.7 works best for our model
\n ():\n \n label_length = tf.reduce_sum(tf.cast(labels != pad_token_idx, tf.int32), axis=-)\n logit_length = tf.ones(tf.shape(logits)[], dtype=tf.int32) * tf.shape(logits)[]\n ctc_loss = tf.nn.ctc_loss(\n labels=labels,\n logits=logits,\n label_length=label_length,\n logit_length=logit_length,\n blank_index=pad_token_idx,\n logits_time_major=\n )\n ctc_loss = tf.reduce_mean(ctc_loss)\n\n \n kl_inp = tf.nn.softmax(logits)\n\n \n kl_tar = tf.fill(tf.shape(logits), / num_classes)\n\n \n kldiv_loss = (tf.keras.losses.KLDivergence(tf.keras.losses.Reduction.NONE)(kl_tar, kl_inp) \n + tf.keras.losses.KLDivergence(tf.keras.losses.Reduction.NONE)(kl_inp, kl_tar))/\n\n kldiv_loss = tf.reduce_mean(kldiv_loss)\n\n \n loss = ( - weight) * ctc_loss + weight * kldiv_loss\n loss\nDuring our experimentation, we found a significant insight: training solely on the supplemental dataset yielded a LB score of 0.367. This highlighted the potential value embedded within the supplemental data.
\nWe decided to first pre-train our model on the supplemental dataset, then loaded these pre-trained weights to further train the model on the main dataset. We tested multiple epochs: 40, 60, and 80 for pre-train. 80 epochs works best and give a improvement of 0.013 on the LB score.
\nAugmentation:
\nIn 2nd place solution, @hoyso48 mention that some augmentation didn't improve the model during short epoch training but in longer training would have the effect. This could applied to our case as well but we haven't verified yet.
Switching to GIC-CTC:
\nhttps://www.kaggle.com/code/wuhongrui/aslfr-gcs-path
\nhttps://www.kaggle.com/wuhongrui/aslfr-training
\nhttps://www.kaggle.com/wuhongrui/ctc-final-inference-notebook
Training notebook is written specific for training on Google Colab. Frist use gcs-path notebook check the path in training notebook. Set USE_SUPPLY = True first to train supply set, then set USE_VAL = True to load pre-train weight and train main data set.
\nTo use it on Kaggle TPU, change the file address to corresponding Kaggle input address and you NEED to modify the CTC loss. Check ASLFR CTC on TPU
", "rawMarkdown": "## Acknowledgements\n\nFirstly, I'd like to extend my deepest gratitude to my teammates:\n- @chellyfan\n- @chuansj\n\nTheir enormous effort has been instrumental in this competition. \n\nI'd like also thanks to the authors of following notebooks. Our success wouldn't have been possible without the insights and techniques we adopted from these notebooks. A big thanks to you.\n\n- **@irohith** for his excellent preprocessing and training notebooks:\n - [ASLFR CTC based on prev comp 1st place](https://www.kaggle.com/code/irohith/aslfr-ctc-based-on-prev-comp-1st-place)\n - [ASLFR preprocess dataset TFRecords mean std](https://www.kaggle.com/code/irohith/aslfr-preprocess-dataset-tfrecords-mean-std)\n\n- **@hoyso48** for his 1st place solution from a previous competition:\n - [1st Place Solution Training](https://www.kaggle.com/code/hoyso48/1st-place-solution-training)\n\n- **@greysnow** for the CTC on TPU notebook:\n - [ASLFR CTC on TPU](https://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu)\n\n- **@royalacecat** for the notebook on more blocks with quantization:\n - [The deeper, the better](https://www.kaggle.com/code/royalacecat/the-deeper-the-better)\n\n## TL;DR\n\nOur solution is based on the CTC open source notebook. Here are the key improvements we introduced (all scores are public LB):\n\n- Only filter **full NaN frames** when generating tfrecord files on data preprocessing. (0.687->0.705)\n- Adding more numbers of block in the model as described in [The deeper, the better](https://www.kaggle.com/code/royalacecat/the-deeper-the-better). (0.705->0.716)\n- **Increased frame length** from 128 to 256. (0.716->0.746)\n- **Enhanced feature selection** by adding more face and pose indices, increase from 96 to 128 indices. (0.746->0.763)\n- Incorporated **CTC loss with label smoothing** for added regularization. (0.763 ->0.766)\n- **Pretrained our model** on the supplemental dataset. Pre train 80 epochs on supply set then 80 epochs on main dataset. (0.766->0.777)\n- Adopted the **motion feature** technique from a previous competition. (0.777->0.781)\n- Increase training epochs on main dataset from 80 to 120 (0.781->0.782)\n\n## Data Preprocessing\n\nReferring to @irohith's preprocessing notebook, we made changes by adding more face and pose indices. We selected the new indices based on their variance — higher variance indicates more information.\n\nInitially, the indices were:\n\n```python\nLIP = [\n 61, 185, 40, 39, 37, 0, 267, 269, 270, 409,\n 291, 146, 91, 181, 84, 17, 314, 405, 321, 375,\n 78, 191, 80, 81, 82, 13, 312, 311, 310, 415,\n 95, 88, 178, 87, 14, 317, 402, 318, 324, 308,\n]\nLPOSE = [13, 15, 17, 19, 21]\nRPOSE = [14, 16, 18, 20, 22]\nPOSE = LPOSE + RPOSE\n```\nWe then updated them to:\n\n```python\n\nLIP = [\n 61, 185, 40, 39, 37, 0, 267, 269, 270, 409,\n 291, 146, 91, 181, 84, 17, 314, 405, 321, 375,\n 78, 191, 80, 81, 82, 13, 312, 311, 310, 415,\n 95, 88, 178, 87, 14, 317, 402, 318, 324, 308,\n]\n\n# New face_id indices\nface_id = [454,356,323,361,389,288,251,264,447,366,368,\n 401,397,435,284,301,372,345,383,367,365,352,433,\n 376,298,265,93,234,300,132,340,353,127]\n\n# Append new face_id indices to LIP if not already present\nfor k in face_id:\n if k not in LIP:\n LIP.append(k)\n\n# Trim the LIP list\nl = len(LIP)\nLIP = LIP[:int(l - l/4)]\n\n# Updated POSE indices\nLPOSE = [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31]\nRPOSE = [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30]\nPOSE = LPOSE + RPOSE\n```\nfilter out full Nan frame\n\norigin notebook:\n\n```python\nhand = tf.concat([rhand, lhand], axis=1)\nhand = tf.where(tf.math.is_nan(hand), 0.0, hand)\nmask = tf.math.not_equal(tf.reduce_sum(hand, axis=[1, 2]), 0.0)\n```\nupdate to \n\n```python\nhand = tf.concat([rhand, lhand,lip,rpose,lpose], axis=1)\nhand = tf.where(tf.math.is_nan(hand), 0.0, hand)\nmask = tf.math.not_equal(tf.reduce_sum(hand, axis=[1, 2]), 0.0)\n```\nThose new indices along with 256 frame length gives us about 0.02 LB improvement!\n\n## Motion feature\nsee [1st Place Solution Training](https://www.kaggle.com/code/hoyso48/1st-place-solution-training)\n\n\n## CTC Loss with Label Smoothing\nTo enhance our model's stability during training, we implemented a CTC loss with an added regularization component. By combining the CTC loss with a Kullback-Leibler (KL) divergence, we introduced label smoothing to the model. Smooth weight = 0.7 works best for our model\n```python\ndef smooth_ctc_loss(labels, logits, num_classes = 60 , blank=0, weight=0.7):\n # Compute CTC Loss\n label_length = tf.reduce_sum(tf.cast(labels != pad_token_idx, tf.int32), axis=-1)\n logit_length = tf.ones(tf.shape(logits)[0], dtype=tf.int32) * tf.shape(logits)[1]\n ctc_loss = tf.nn.ctc_loss(\n labels=labels,\n logits=logits,\n label_length=label_length,\n logit_length=logit_length,\n blank_index=pad_token_idx,\n logits_time_major=False\n )\n ctc_loss = tf.reduce_mean(ctc_loss)\n\n # Compute KL Divergence Loss\n kl_inp = tf.nn.softmax(logits)\n\n # Create the target distribution\n kl_tar = tf.fill(tf.shape(logits), 1. / num_classes)\n\n # Compute the KL divergence\n kldiv_loss = (tf.keras.losses.KLDivergence(tf.keras.losses.Reduction.NONE)(kl_tar, kl_inp) \n + tf.keras.losses.KLDivergence(tf.keras.losses.Reduction.NONE)(kl_inp, kl_tar))/2.0\n\n kldiv_loss = tf.reduce_mean(kldiv_loss)\n\n # Combined Loss\n loss = (1. - weight) * ctc_loss + weight * kldiv_loss\n return loss\n```\n## Pre-training the Model on the Supplemental Dataset\n\nDuring our experimentation, we found a significant insight: training solely on the **supplemental dataset** yielded a LB score of 0.367. This highlighted the potential value embedded within the supplemental data.\n\nWe decided to first pre-train our model on the supplemental dataset, then loaded these pre-trained weights to further train the model on the main dataset. We tested multiple epochs: 40, 60, and 80 for pre-train. 80 epochs works best and give a improvement of 0.013 on the LB score.\n\n## What We Tried, But Didn't Work\n\n1. **Augmentation**:\n - **Random Affine Transformation**\n - **Random Part Removal**: We tried dropping certain parts (like lip, lpose, rpose, lhand, rhand) by setting them to NaN with a 5% probability. \n - **Feature Switch for Phrases**: switch features for phrases within the same category.\n\n In 2nd place solution, @hoyso48 mention that some augmentation didn't improve the model during short epoch training but in longer training would have the effect. This could applied to our case as well but we haven't verified yet.\n\n2. **Switching to GIC-CTC**:\n - **Reference Paper**: [Improving CTC-Based ASR Models With Gated Interlayer Collaboration](https://ieeexplore.ieee.org/document/10094820)\n - Increasing parameters led to our model exceeding the 5-hour inference time limit.\n\n\n## What We could Imporve\n1. long epochs training. We only train 120 epochs for final submission, which is much less compare to other top place solution (usually 300-500 epochs). We already saw a 0.001 LB improve from 80 epochs to 120 epochs on the final day, but it's too late\n2. Adjust the leaning rate and weight decay more carefully. We simply use the cosine decay lr scheduler in the public notebook and set weight decay to 0.05\n3. Utilized AWP for longer epoch training\n\n## Notebook:\n \nhttps://www.kaggle.com/code/wuhongrui/aslfr-gcs-path\nhttps://www.kaggle.com/wuhongrui/aslfr-training\nhttps://www.kaggle.com/wuhongrui/ctc-final-inference-notebook\n\nTraining notebook is written specific for training on Google Colab. Frist use gcs-path notebook check the path in training notebook. Set USE_SUPPLY = True first to train supply set, then set USE_VAL = True to load pre-train weight and train main data set.\n\nTo use it on Kaggle TPU, change the file address to corresponding Kaggle input address and you NEED to modify the CTC loss. Check [ASLFR CTC on TPU](https://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu)\n\n\n\n\n\n\n\n\n ", "votes": 6 }, { "id": 2410646, "postDate": "2023-08-27T06:19:15.497Z", "content": "Thank you for sharing.
\n\n\nWe decided to first pre-train our model on the supplemental dataset, then loaded these pre-trained weights to further train the model on the main dataset. We tested multiple epochs: 40, 60, and 80 for pre-train. 80 epochs works best and give a improvement of 0.013 on the LB score.
\n
We saw this too for early models. But as soon as models became better, pretraining on supplemental did not have effect anymore. Maybe also because our better models are trained longer (just using cutmix doubles the number of optimal epochs from 200 to 400 for us) and weights initialization is not that important then.
", "rawMarkdown": "Thank you for sharing.\n\n>We decided to first pre-train our model on the supplemental dataset, then loaded these pre-trained weights to further train the model on the main dataset. We tested multiple epochs: 40, 60, and 80 for pre-train. 80 epochs works best and give a improvement of 0.013 on the LB score.\n\nWe saw this too for early models. But as soon as models became better, pretraining on supplemental did not have effect anymore. Maybe also because our better models are trained longer (just using cutmix doubles the number of optimal epochs from 200 to 400 for us) and weights initialization is not that important then.", "votes": 2, "replies": [ { "id": 2410669, "postDate": "2023-08-27T06:38:27.903Z", "content": "Thanks for your great insight! It's interesting to see the impact of pre-training and the number of epochs on your model's performance. We overlooked importance of the long epoch training in this competition, and as you mentioned, the weights initialization maybe is less significant for long epoch.
\nCongratulation again for your amazing solution of 1st place in this competition !
", "rawMarkdown": "Thanks for your great insight! It's interesting to see the impact of pre-training and the number of epochs on your model's performance. We overlooked importance of the long epoch training in this competition, and as you mentioned, the weights initialization maybe is less significant for long epoch.\n\nCongratulation again for your amazing solution of 1st place in this competition !", "votes": 2 } ] }, { "id": 2412159, "postDate": "2023-08-28T05:51:35.923Z", "content": "Congratulations. Just improving the score with long no of epochs can backfire if there is overfitting or the model encounter local minima.
", "rawMarkdown": "Congratulations. Just improving the score with long no of epochs can backfire if there is overfitting or the model encounter local minima. " } ], "comments": [ { "id": 2410646, "author_name": "Dieter", "author_url": "", "post_date": "2023-08-27T06:19:15.497000", "content": "Thank you for sharing.
\n\n\nWe decided to first pre-train our model on the supplemental dataset, then loaded these pre-trained weights to further train the model on the main dataset. We tested multiple epochs: 40, 60, and 80 for pre-train. 80 epochs works best and give a improvement of 0.013 on the LB score.
\n
We saw this too for early models. But as soon as models became better, pretraining on supplemental did not have effect anymore. Maybe also because our better models are trained longer (just using cutmix doubles the number of optimal epochs from 200 to 400 for us) and weights initialization is not that important then.
", "votes": 2, "replies": [ { "id": 2410669, "author_name": "HW", "author_url": "", "post_date": "2023-08-27T06:38:27.903000", "content": "Thanks for your great insight! It's interesting to see the impact of pre-training and the number of epochs on your model's performance. We overlooked importance of the long epoch training in this competition, and as you mentioned, the weights initialization maybe is less significant for long epoch.
\nCongratulation again for your amazing solution of 1st place in this competition !
", "votes": 2, "replies": [] } ] }, { "id": 2412159, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-08-28T05:51:35.923000", "content": "Congratulations. Just improving the score with long no of epochs can backfire if there is overfitting or the model encounter local minima.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2410560": "## Acknowledgements\n\nFirstly, I'd like to extend my deepest gratitude to my teammates:\n- @chellyfan\n- @chuansj\n\nTheir enormous effort has been instrumental in this competition. \n\nI'd like also thanks to the authors of following notebooks. Our success wouldn't have been possible without the insights and techniques we adopted from these notebooks. A big thanks to you.\n\n- **@irohith** for his excellent preprocessing and training notebooks:\n - [ASLFR CTC based on prev comp 1st place](https://www.kaggle.com/code/irohith/aslfr-ctc-based-on-prev-comp-1st-place)\n - [ASLFR preprocess dataset TFRecords mean std](https://www.kaggle.com/code/irohith/aslfr-preprocess-dataset-tfrecords-mean-std)\n\n- **@hoyso48** for his 1st place solution from a previous competition:\n - [1st Place Solution Training](https://www.kaggle.com/code/hoyso48/1st-place-solution-training)\n\n- **@greysnow** for the CTC on TPU notebook:\n - [ASLFR CTC on TPU](https://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu)\n\n- **@royalacecat** for the notebook on more blocks with quantization:\n - [The deeper, the better](https://www.kaggle.com/code/royalacecat/the-deeper-the-better)\n\n## TL;DR\n\nOur solution is based on the CTC open source notebook. Here are the key improvements we introduced (all scores are public LB):\n\n- Only filter **full NaN frames** when generating tfrecord files on data preprocessing. (0.687->0.705)\n- Adding more numbers of block in the model as described in [The deeper, the better](https://www.kaggle.com/code/royalacecat/the-deeper-the-better). (0.705->0.716)\n- **Increased frame length** from 128 to 256. (0.716->0.746)\n- **Enhanced feature selection** by adding more face and pose indices, increase from 96 to 128 indices. (0.746->0.763)\n- Incorporated **CTC loss with label smoothing** for added regularization. (0.763 ->0.766)\n- **Pretrained our model** on the supplemental dataset. Pre train 80 epochs on supply set then 80 epochs on main dataset. (0.766->0.777)\n- Adopted the **motion feature** technique from a previous competition. (0.777->0.781)\n- Increase training epochs on main dataset from 80 to 120 (0.781->0.782)\n\n## Data Preprocessing\n\nReferring to @irohith's preprocessing notebook, we made changes by adding more face and pose indices. We selected the new indices based on their variance — higher variance indicates more information.\n\nInitially, the indices were:\n\n```python\nLIP = [\n 61, 185, 40, 39, 37, 0, 267, 269, 270, 409,\n 291, 146, 91, 181, 84, 17, 314, 405, 321, 375,\n 78, 191, 80, 81, 82, 13, 312, 311, 310, 415,\n 95, 88, 178, 87, 14, 317, 402, 318, 324, 308,\n]\nLPOSE = [13, 15, 17, 19, 21]\nRPOSE = [14, 16, 18, 20, 22]\nPOSE = LPOSE + RPOSE\n```\nWe then updated them to:\n\n```python\n\nLIP = [\n 61, 185, 40, 39, 37, 0, 267, 269, 270, 409,\n 291, 146, 91, 181, 84, 17, 314, 405, 321, 375,\n 78, 191, 80, 81, 82, 13, 312, 311, 310, 415,\n 95, 88, 178, 87, 14, 317, 402, 318, 324, 308,\n]\n\n# New face_id indices\nface_id = [454,356,323,361,389,288,251,264,447,366,368,\n 401,397,435,284,301,372,345,383,367,365,352,433,\n 376,298,265,93,234,300,132,340,353,127]\n\n# Append new face_id indices to LIP if not already present\nfor k in face_id:\n if k not in LIP:\n LIP.append(k)\n\n# Trim the LIP list\nl = len(LIP)\nLIP = LIP[:int(l - l/4)]\n\n# Updated POSE indices\nLPOSE = [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31]\nRPOSE = [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30]\nPOSE = LPOSE + RPOSE\n```\nfilter out full Nan frame\n\norigin notebook:\n\n```python\nhand = tf.concat([rhand, lhand], axis=1)\nhand = tf.where(tf.math.is_nan(hand), 0.0, hand)\nmask = tf.math.not_equal(tf.reduce_sum(hand, axis=[1, 2]), 0.0)\n```\nupdate to \n\n```python\nhand = tf.concat([rhand, lhand,lip,rpose,lpose], axis=1)\nhand = tf.where(tf.math.is_nan(hand), 0.0, hand)\nmask = tf.math.not_equal(tf.reduce_sum(hand, axis=[1, 2]), 0.0)\n```\nThose new indices along with 256 frame length gives us about 0.02 LB improvement!\n\n## Motion feature\nsee [1st Place Solution Training](https://www.kaggle.com/code/hoyso48/1st-place-solution-training)\n\n\n## CTC Loss with Label Smoothing\nTo enhance our model's stability during training, we implemented a CTC loss with an added regularization component. By combining the CTC loss with a Kullback-Leibler (KL) divergence, we introduced label smoothing to the model. Smooth weight = 0.7 works best for our model\n```python\ndef smooth_ctc_loss(labels, logits, num_classes = 60 , blank=0, weight=0.7):\n # Compute CTC Loss\n label_length = tf.reduce_sum(tf.cast(labels != pad_token_idx, tf.int32), axis=-1)\n logit_length = tf.ones(tf.shape(logits)[0], dtype=tf.int32) * tf.shape(logits)[1]\n ctc_loss = tf.nn.ctc_loss(\n labels=labels,\n logits=logits,\n label_length=label_length,\n logit_length=logit_length,\n blank_index=pad_token_idx,\n logits_time_major=False\n )\n ctc_loss = tf.reduce_mean(ctc_loss)\n\n # Compute KL Divergence Loss\n kl_inp = tf.nn.softmax(logits)\n\n # Create the target distribution\n kl_tar = tf.fill(tf.shape(logits), 1. / num_classes)\n\n # Compute the KL divergence\n kldiv_loss = (tf.keras.losses.KLDivergence(tf.keras.losses.Reduction.NONE)(kl_tar, kl_inp) \n + tf.keras.losses.KLDivergence(tf.keras.losses.Reduction.NONE)(kl_inp, kl_tar))/2.0\n\n kldiv_loss = tf.reduce_mean(kldiv_loss)\n\n # Combined Loss\n loss = (1. - weight) * ctc_loss + weight * kldiv_loss\n return loss\n```\n## Pre-training the Model on the Supplemental Dataset\n\nDuring our experimentation, we found a significant insight: training solely on the **supplemental dataset** yielded a LB score of 0.367. This highlighted the potential value embedded within the supplemental data.\n\nWe decided to first pre-train our model on the supplemental dataset, then loaded these pre-trained weights to further train the model on the main dataset. We tested multiple epochs: 40, 60, and 80 for pre-train. 80 epochs works best and give a improvement of 0.013 on the LB score.\n\n## What We Tried, But Didn't Work\n\n1. **Augmentation**:\n - **Random Affine Transformation**\n - **Random Part Removal**: We tried dropping certain parts (like lip, lpose, rpose, lhand, rhand) by setting them to NaN with a 5% probability. \n - **Feature Switch for Phrases**: switch features for phrases within the same category.\n\n In 2nd place solution, @hoyso48 mention that some augmentation didn't improve the model during short epoch training but in longer training would have the effect. This could applied to our case as well but we haven't verified yet.\n\n2. **Switching to GIC-CTC**:\n - **Reference Paper**: [Improving CTC-Based ASR Models With Gated Interlayer Collaboration](https://ieeexplore.ieee.org/document/10094820)\n - Increasing parameters led to our model exceeding the 5-hour inference time limit.\n\n\n## What We could Imporve\n1. long epochs training. We only train 120 epochs for final submission, which is much less compare to other top place solution (usually 300-500 epochs). We already saw a 0.001 LB improve from 80 epochs to 120 epochs on the final day, but it's too late\n2. Adjust the leaning rate and weight decay more carefully. We simply use the cosine decay lr scheduler in the public notebook and set weight decay to 0.05\n3. Utilized AWP for longer epoch training\n\n## Notebook:\n \nhttps://www.kaggle.com/code/wuhongrui/aslfr-gcs-path\nhttps://www.kaggle.com/wuhongrui/aslfr-training\nhttps://www.kaggle.com/wuhongrui/ctc-final-inference-notebook\n\nTraining notebook is written specific for training on Google Colab. Frist use gcs-path notebook check the path in training notebook. Set USE_SUPPLY = True first to train supply set, then set USE_VAL = True to load pre-train weight and train main data set.\n\nTo use it on Kaggle TPU, change the file address to corresponding Kaggle input address and you NEED to modify the CTC loss. Check [ASLFR CTC on TPU](https://www.kaggle.com/code/shlomoron/aslfr-ctc-on-tpu)\n\n\n\n\n\n\n\n\n ", "2410646": "Thank you for sharing.\n\n>We decided to first pre-train our model on the supplemental dataset, then loaded these pre-trained weights to further train the model on the main dataset. We tested multiple epochs: 40, 60, and 80 for pre-train. 80 epochs works best and give a improvement of 0.013 on the LB score.\n\nWe saw this too for early models. But as soon as models became better, pretraining on supplemental did not have effect anymore. Maybe also because our better models are trained longer (just using cutmix doubles the number of optimal epochs from 200 to 400 for us) and weights initialization is not that important then.", "2412159": "Congratulations. Just improving the score with long no of epochs can backfire if there is overfitting or the model encounter local minima. " } }