My best-performing model (config)is based on the 1st place solution of the previous competition by @hoyso48. I trained the model using vanilla CTC Loss, and used greedy decoding for inference.
\nLonger input frames length: A longer input frame length of 384 performed better than shorter length of 256.
Deeper/larger model: My best-performing model has 7 blocks with hidden dimension of 256. It has 9.5M parameters.
Pose keypoints: Adding pose information was helpful. Best-performing model uses pose+hands+lips+eyes.
CNN+MHSA: CNN+MHSA model > Only CNN Model>Only MHSA model
Strong augmentation: My model was performing well based on local CV, and public LB, with strong correlations between the two.But when I tested the model using this Gradio app using my webcam, it could not recognize my signs, so I had to use strong data augmentations to get the model to work. Especially temporal mask helped because I sign slower than the pros. These augmentations also boosted the public LB score by +0.006.
\nflip_lr_probability=0.5
\nrandom_affine_probability=0.75
\nfreeze_probability=0.5
\ntemporal_mask_probability=0.75
\ntemporal_mask_range=(0.2,0.4)
concat augmentation: Randomly concatenate two short landmark sequences, as well as their labels. This improved public LB by +0.008. I applied this augmentation to 40% of all training samples.
transformer-encoder+decoder: Transformer endoder-decoder model with cross-entropy loss.
transformer-decoder: CNN+MHSA model and Transformer-like decoder with cross-entropy loss.
Causal-masking in self-attention: removing causal masking performed better than using causal mask.
Attention-span in self-attention: I thought that there would not be long-term dependency between frames for this task, so I tried to reduce the attention span of self-attention. Although there was no performance degradation, it did not boost performance either.
Focal loss: I tried the CTC Focal loss based on this repo, but there were no gains.
Erase landmarks augmentation: Randomly erase landmarks except the hand landmarks, to make the model more robust to mediapipe's detection errors.
Originally I was using pytorch, but I switched to Tensorflow because I faced several issues when converting the pyTorch model to tfLite. However, training with CTC Loss was upto 10x slower in tensorflow than pytorch. Looking back, I think I should have put more effort into fixing the model conversion issue, as I would have been able to perform more experiments.
\nCongratulations to the winning teams. I would also like to thank Google for hosting this competition, it was a valuable opporunity to learn many new things. Also, I would like to thank everyone who shared their notebooks and unique ideas.
", "messages": [ { "id": 2409214, "postDate": "2023-08-26T05:29:18.700Z", "content": "My best-performing model (config)is based on the 1st place solution of the previous competition by @hoyso48. I trained the model using vanilla CTC Loss, and used greedy decoding for inference.
\nLonger input frames length: A longer input frame length of 384 performed better than shorter length of 256.
Deeper/larger model: My best-performing model has 7 blocks with hidden dimension of 256. It has 9.5M parameters.
Pose keypoints: Adding pose information was helpful. Best-performing model uses pose+hands+lips+eyes.
CNN+MHSA: CNN+MHSA model > Only CNN Model>Only MHSA model
Strong augmentation: My model was performing well based on local CV, and public LB, with strong correlations between the two.But when I tested the model using this Gradio app using my webcam, it could not recognize my signs, so I had to use strong data augmentations to get the model to work. Especially temporal mask helped because I sign slower than the pros. These augmentations also boosted the public LB score by +0.006.
\nflip_lr_probability=0.5
\nrandom_affine_probability=0.75
\nfreeze_probability=0.5
\ntemporal_mask_probability=0.75
\ntemporal_mask_range=(0.2,0.4)
concat augmentation: Randomly concatenate two short landmark sequences, as well as their labels. This improved public LB by +0.008. I applied this augmentation to 40% of all training samples.
transformer-encoder+decoder: Transformer endoder-decoder model with cross-entropy loss.
transformer-decoder: CNN+MHSA model and Transformer-like decoder with cross-entropy loss.
Causal-masking in self-attention: removing causal masking performed better than using causal mask.
Attention-span in self-attention: I thought that there would not be long-term dependency between frames for this task, so I tried to reduce the attention span of self-attention. Although there was no performance degradation, it did not boost performance either.
Focal loss: I tried the CTC Focal loss based on this repo, but there were no gains.
Erase landmarks augmentation: Randomly erase landmarks except the hand landmarks, to make the model more robust to mediapipe's detection errors.
Originally I was using pytorch, but I switched to Tensorflow because I faced several issues when converting the pyTorch model to tfLite. However, training with CTC Loss was upto 10x slower in tensorflow than pytorch. Looking back, I think I should have put more effort into fixing the model conversion issue, as I would have been able to perform more experiments.
\nCongratulations to the winning teams. I would also like to thank Google for hosting this competition, it was a valuable opporunity to learn many new things. Also, I would like to thank everyone who shared their notebooks and unique ideas.
", "rawMarkdown": "[My best-performing model](https://github.com/SamratThapa120/sign-language-finger-spelling/blob/master/signet/models/feature_extractor_downsampled.py) ([config](https://github.com/SamratThapa120/sign-language-finger-spelling/blob/master/signet/configs/ctc_loss_with_downsampled_deploy_concataug.py))is based on the [1st place solution of the previous competition](https://www.kaggle.com/code/hoyso48/1st-place-solution-training/notebook) by @hoyso48. I trained the model using vanilla CTC Loss, and used greedy decoding for inference. \n\n### Things that worked for me:\n- **Longer input frames length:** A longer input frame length of 384 performed better than shorter length of 256. \n- **Deeper/larger model:** My best-performing model has 7 blocks with hidden dimension of 256. It has 9.5M parameters.\n- **Pose keypoints:** Adding pose information was helpful. Best-performing model uses pose+hands+lips+eyes.\n- **CNN+MHSA:** CNN+MHSA model > Only CNN Model>Only MHSA model\n- **Strong augmentation:** My model was performing well based on local CV, and public LB, with strong correlations between the two.But when I tested the model using this [Gradio app](https://github.com/SamratThapa120/gradio-ASL-fingerspelling-recognition) using my webcam, it could not recognize my signs, so I had to use strong data augmentations to get the model to work. Especially temporal mask helped because I sign slower than the pros. These augmentations also boosted the public LB score by +0.006.\n\n flip_lr_probability=0.5\n random_affine_probability=0.75\n freeze_probability=0.5\n temporal_mask_probability=0.75\n temporal_mask_range=(0.2,0.4)\n\n- **concat augmentation:** Randomly concatenate two short landmark sequences, as well as their labels. This improved public LB by +0.008. I applied this augmentation to 40% of all training samples.\n\n### Here are other things I tried, that did not contribute to the best-performing model:\n\n- **transformer-encoder+decoder:** Transformer endoder-decoder model with cross-entropy loss.\n\n- **transformer-decoder:** CNN+MHSA model and Transformer-like decoder with cross-entropy loss.\n\n- **Causal-masking in self-attention:** removing causal masking performed better than using causal mask.\n\n- **Attention-span in self-attention:** I thought that there would not be long-term dependency between frames for this task, so I tried to reduce the attention span of self-attention. Although there was no performance degradation, it did not boost performance either.\n\n- **Focal loss**: I tried the CTC Focal loss based on [this](https://github.com/TeaPoly/CTC-OptimizedLoss/blob/main/ctc_focal_loss.py) repo, but there were no gains.\n\n- **Erase landmarks augmentation**: Randomly erase landmarks except the hand landmarks, to make the model more robust to mediapipe's detection errors.\n\nOriginally I was using [pytorch](https://github.com/SamratThapa120/sign-language-finger-spelling/tree/pytorch), but I switched to Tensorflow because I faced several issues when converting the pyTorch model to tfLite. However, training with CTC Loss was upto 10x slower in tensorflow than pytorch. Looking back, I think I should have put more effort into fixing the model conversion issue, as I would have been able to perform more experiments. \n\nCongratulations to the winning teams. I would also like to thank Google for hosting this competition, it was a valuable opporunity to learn many new things. Also, I would like to thank everyone who shared their notebooks and unique ideas.\n\n", "votes": 4 }, { "id": 2409236, "postDate": "2023-08-26T05:45:48.377Z", "content": "Thank you for sharing. Great to see that you used concat augmentation. Could you quickly explain how freeze augmentation works?
", "rawMarkdown": "Thank you for sharing. Great to see that you used concat augmentation. Could you quickly explain how freeze augmentation works?", "votes": 1, "replies": [ { "id": 2409261, "postDate": "2023-08-26T06:06:45.150Z", "content": "Freeze augmentation tries to replicate frozen frames in a video. Randomly select some frames and replicate them without changing their order in the sequence. For example [a,b,c,d,e] will get converted to something like [a,a,a,b,b,c,d,e,e].
\n", "rawMarkdown": "Freeze augmentation tries to replicate frozen frames in a video. Randomly select some frames and replicate them without changing their order in the sequence. For example [a,b,c,d,e] will get converted to something like [a,a,a,b,b,c,d,e,e]. \n\n[Here is the code](https://github.com/SamratThapa120/sign-language-finger-spelling/blob/2bbc9a4c51622d3553ab9be17715ca51332c8797/signet/dataset/transforms.py#L215)", "votes": 1 } ] }, { "id": 2410124, "postDate": "2023-08-26T16:41:34.687Z", "content": "Congratulations. Thanks for sharing the techniques useful for higher scores.
\nAre the augmentations random or additional data set is generated.
The augmentations are applied randomly during training
", "rawMarkdown": "The augmentations are applied randomly during training" } ] }, { "id": 2414029, "postDate": "2023-08-29T10:22:27.170Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2409236, "author_name": "Dieter", "author_url": "", "post_date": "2023-08-26T05:45:48.377000", "content": "Thank you for sharing. Great to see that you used concat augmentation. Could you quickly explain how freeze augmentation works?
", "votes": 1, "replies": [ { "id": 2409261, "author_name": "Samrat Thapa", "author_url": "", "post_date": "2023-08-26T06:06:45.150000", "content": "Freeze augmentation tries to replicate frozen frames in a video. Randomly select some frames and replicate them without changing their order in the sequence. For example [a,b,c,d,e] will get converted to something like [a,a,a,b,b,c,d,e,e].
\n", "votes": 1, "replies": [] } ] }, { "id": 2410124, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-08-26T16:41:34.687000", "content": "Congratulations. Thanks for sharing the techniques useful for higher scores.
\nAre the augmentations random or additional data set is generated.
The augmentations are applied randomly during training
", "votes": 0, "replies": [] } ] }, { "id": 2414029, "author_name": "", "author_url": "", "post_date": "2023-08-29T10:22:27.170000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2409214": "[My best-performing model](https://github.com/SamratThapa120/sign-language-finger-spelling/blob/master/signet/models/feature_extractor_downsampled.py) ([config](https://github.com/SamratThapa120/sign-language-finger-spelling/blob/master/signet/configs/ctc_loss_with_downsampled_deploy_concataug.py))is based on the [1st place solution of the previous competition](https://www.kaggle.com/code/hoyso48/1st-place-solution-training/notebook) by @hoyso48. I trained the model using vanilla CTC Loss, and used greedy decoding for inference. \n\n### Things that worked for me:\n- **Longer input frames length:** A longer input frame length of 384 performed better than shorter length of 256. \n- **Deeper/larger model:** My best-performing model has 7 blocks with hidden dimension of 256. It has 9.5M parameters.\n- **Pose keypoints:** Adding pose information was helpful. Best-performing model uses pose+hands+lips+eyes.\n- **CNN+MHSA:** CNN+MHSA model > Only CNN Model>Only MHSA model\n- **Strong augmentation:** My model was performing well based on local CV, and public LB, with strong correlations between the two.But when I tested the model using this [Gradio app](https://github.com/SamratThapa120/gradio-ASL-fingerspelling-recognition) using my webcam, it could not recognize my signs, so I had to use strong data augmentations to get the model to work. Especially temporal mask helped because I sign slower than the pros. These augmentations also boosted the public LB score by +0.006.\n\n flip_lr_probability=0.5\n random_affine_probability=0.75\n freeze_probability=0.5\n temporal_mask_probability=0.75\n temporal_mask_range=(0.2,0.4)\n\n- **concat augmentation:** Randomly concatenate two short landmark sequences, as well as their labels. This improved public LB by +0.008. I applied this augmentation to 40% of all training samples.\n\n### Here are other things I tried, that did not contribute to the best-performing model:\n\n- **transformer-encoder+decoder:** Transformer endoder-decoder model with cross-entropy loss.\n\n- **transformer-decoder:** CNN+MHSA model and Transformer-like decoder with cross-entropy loss.\n\n- **Causal-masking in self-attention:** removing causal masking performed better than using causal mask.\n\n- **Attention-span in self-attention:** I thought that there would not be long-term dependency between frames for this task, so I tried to reduce the attention span of self-attention. Although there was no performance degradation, it did not boost performance either.\n\n- **Focal loss**: I tried the CTC Focal loss based on [this](https://github.com/TeaPoly/CTC-OptimizedLoss/blob/main/ctc_focal_loss.py) repo, but there were no gains.\n\n- **Erase landmarks augmentation**: Randomly erase landmarks except the hand landmarks, to make the model more robust to mediapipe's detection errors.\n\nOriginally I was using [pytorch](https://github.com/SamratThapa120/sign-language-finger-spelling/tree/pytorch), but I switched to Tensorflow because I faced several issues when converting the pyTorch model to tfLite. However, training with CTC Loss was upto 10x slower in tensorflow than pytorch. Looking back, I think I should have put more effort into fixing the model conversion issue, as I would have been able to perform more experiments. \n\nCongratulations to the winning teams. I would also like to thank Google for hosting this competition, it was a valuable opporunity to learn many new things. Also, I would like to thank everyone who shared their notebooks and unique ideas.\n\n", "2409236": "Thank you for sharing. Great to see that you used concat augmentation. Could you quickly explain how freeze augmentation works?", "2410124": "Congratulations. Thanks for sharing the techniques useful for higher scores.\nAre the augmentations random or additional data set is generated. ", "2414029": "" } }