Thanks to the Kaggle team and organizers for this challenging competition. Huge props to my teammates @wuliaokaola @bacterio, they worked very hard!! We have learned a lot and now we are exhausted both computationally and physically 😃, just like many of you. This solution is developed within 15 days and it's far away from the top winning solutions. But hopefully there's something helpful in this!
\nEfficientnet B1 for image level, Efficientnet 3D CNN + Resnet 3D for study level. CNN + Transformer for image and study level.
Used TFRecords by the organizers and windows function by @vaillant.
Applied mask to hide left/right side of the image
Optimized blending weights based on OOF calculation.
We started the competition about 15 days ago.
Trained Efficientnet B0 on image level and blended with the mean predictions by @osciiart, aka the public notebook scored 0.325. 0.4 * mean prediction + 0.6 * Efficient B0 along with the mean prediction for study levels. This gives us LB: 0.292.
Developed Resnet3D that scored 0.370 LB. We ran a local validation for this model and found out that this model did well on the study level but poorly on the image level. Hence, for study level, we replaced then mean prediction with our Resnet3D and achieved 0.253 LB.
Improved Efficientnet B0 with masks(masking one half of the image). And achieved 0.248 LB.
The amazing public baseline came out and we were frustrated because it scored 0.233 and it made us panic. We tried to inference that and it took forever so we decided to not incorporate that into our pipeline.
We added @bacterio to our team and blended with our existing results(take the mean) and got 0.226 LB. @bacterio had a CNN + Transformer at that time, which is completely disparate from our approach. It calmed us down a bit and we knew that if we continue improving both image level and study level, we will get better.
We improved by training three Efficientnet B1s with similar techniques and @wuliaokaola came out with a new architecture Efficientnet 3D CNN and we got 0.212 LB.
Oh! We only had two days left. What should we do? We realized that it's quite unrealistic add in another architecture. So we should work on upgrading existing models. We added TTAs to Efficientnet 3D CNN and fine-tuned Efficientnet B1, Resnet 3D, Efficientnet 3D CNN and we got 0.204 LB.
In this section I will show our final models.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5168115%2F27df2eb5da63345c737175b23062be4f%2Fmask.png?generation=1603764000748568&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5168115%2Fb03deaf567cceb73532a54fcdca4d53d%2FScreenshot_2020-10-26%20b1_fold3%20Kaggle.png?generation=1603764244357127&alt=media\")
0.22 image level weighted loss![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5168115%2F1ea1b91bcae244d0c00abda661385556%2FScreenshot_2020-10-26%20rsna%20sped%20cv%201026%20tta.png?generation=1603765140289343&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5168115%2F617169bcd0724022099c5bf12568dbdb%2FScreenshot_2020-10-26%20rsna%20sped%20cv%201026%20tta(1).png?generation=1603765158757751&alt=media\")
0.245 on image level and 0.270 on exam level with 5xTTAs![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5168115%2F6c30952a10b542c019b5745f10c213dd%2FScreenshot_2020-10-26%20rsna%20sped%20cv%201026%20tta(2).png?generation=1603765267593556&alt=media\")
0.19 with weight [0.71, 0.05, 0.07, 0.38, 0.28, 0.78, 0.71, 0.13, 0.45] for each label.Yes, this competition is really heavy on hardwares. We also found the problem of IO bottleneck, i.e. GPU cannot run at full power, as shown below. Here are a list of hardwares we primarily use:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5168115%2F6872f49c2ad156431fdceab9512fbae1%2FScreenshot_2020-10-26%20Slack%20general%20KLLEE.png?generation=1603766768077129&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5168115%2Fd91e7eea26f12e581cf6bf8109853b47%2FScreenshot_2020-10-26%20rsna%20sped%20submit%201026%20tta.png?generation=1603766413299332&alt=media\")
Thanks for reading this solution. Happy kaggling!
", "messages": [ { "id": 1061456, "postDate": "2020-10-27T02:48:21.423Z", "content": "Thanks to the Kaggle team and organizers for this challenging competition. Huge props to my teammates @wuliaokaola @bacterio, they worked very hard!! We have learned a lot and now we are exhausted both computationally and physically 😃, just like many of you. This solution is developed within 15 days and it's far away from the top winning solutions. But hopefully there's something helpful in this!
\nEfficientnet B1 for image level, Efficientnet 3D CNN + Resnet 3D for study level. CNN + Transformer for image and study level.
Used TFRecords by the organizers and windows function by @vaillant.
Applied mask to hide left/right side of the image
Optimized blending weights based on OOF calculation.
We started the competition about 15 days ago.
Trained Efficientnet B0 on image level and blended with the mean predictions by @osciiart, aka the public notebook scored 0.325. 0.4 * mean prediction + 0.6 * Efficient B0 along with the mean prediction for study levels. This gives us LB: 0.292.
Developed Resnet3D that scored 0.370 LB. We ran a local validation for this model and found out that this model did well on the study level but poorly on the image level. Hence, for study level, we replaced then mean prediction with our Resnet3D and achieved 0.253 LB.
Improved Efficientnet B0 with masks(masking one half of the image). And achieved 0.248 LB.
The amazing public baseline came out and we were frustrated because it scored 0.233 and it made us panic. We tried to inference that and it took forever so we decided to not incorporate that into our pipeline.
We added @bacterio to our team and blended with our existing results(take the mean) and got 0.226 LB. @bacterio had a CNN + Transformer at that time, which is completely disparate from our approach. It calmed us down a bit and we knew that if we continue improving both image level and study level, we will get better.
We improved by training three Efficientnet B1s with similar techniques and @wuliaokaola came out with a new architecture Efficientnet 3D CNN and we got 0.212 LB.
Oh! We only had two days left. What should we do? We realized that it's quite unrealistic add in another architecture. So we should work on upgrading existing models. We added TTAs to Efficientnet 3D CNN and fine-tuned Efficientnet B1, Resnet 3D, Efficientnet 3D CNN and we got 0.204 LB.
In this section I will show our final models.
\n0.22 image level weighted loss0.245 on image level and 0.270 on exam level with 5xTTAs0.19 with weight [0.71, 0.05, 0.07, 0.38, 0.28, 0.78, 0.71, 0.13, 0.45] for each label.Yes, this competition is really heavy on hardwares. We also found the problem of IO bottleneck, i.e. GPU cannot run at full power, as shown below. Here are a list of hardwares we primarily use:
\nThanks for reading this solution. Happy kaggling!
", "rawMarkdown": "Thanks to the Kaggle team and organizers for this challenging competition. Huge props to my teammates @wuliaokaola @bacterio, they worked very hard!! We have learned a lot and now we are exhausted both computationally and physically 😃, just like many of you. This solution is developed within 15 days and it's far away from the top winning solutions. But hopefully there's something helpful in this!\n\n## TL; DR\n* **Efficientnet B1** for image level, **Efficientnet 3D CNN** + **Resnet 3D** for study level. **CNN + Transformer** for image and study level.\n\n* Used TFRecords by the organizers and windows function by @vaillant.\n\n* Applied mask to hide left/right side of the image\n\n* Optimized blending weights based on OOF calculation.\n\n## Timeline\n\n* We started the competition about 15 days ago. \n\n* Trained **Efficientnet B0** on image level and blended with the mean predictions by @osciiart, aka the public notebook scored 0.325. 0.4 * mean prediction + 0.6 * Efficient B0 along with the mean prediction for study levels. This gives us LB: `0.292`. \n\n* Developed **Resnet3D** that scored `0.370` LB. We ran a local validation for this model and found out that this model did well on the study level but poorly on the image level. Hence, for study level, we replaced then mean prediction with our Resnet3D and achieved `0.253` LB.\n\n* Improved **Efficientnet B0** with masks(masking one half of the image). And achieved `0.248` LB. \n\n* The amazing public baseline came out and we were frustrated because it scored `0.233` and it made us panic. We tried to inference that and it took forever so we decided to not incorporate that into our pipeline.\n\n* We added @bacterio to our team and blended with our existing results(take the mean) and got `0.226` LB. @bacterio had a **CNN + Transformer** at that time, which is completely disparate from our approach. It calmed us down a bit and we knew that if we continue improving both image level and study level, we will get better.\n\n* We improved by training three **Efficientnet B1**s with similar techniques and @wuliaokaola came out with a new architecture **Efficientnet 3D CNN** and we got `0.212` LB. \n\n* Oh! We only had **two days left**. What should we do? We realized that it's quite unrealistic add in another architecture. So we should work on upgrading existing models. We added TTAs to **Efficientnet 3D CNN** and fine-tuned **Efficientnet B1**, **Resnet 3D**, **Efficientnet 3D CNN** and we got `0.204` LB.\n\n## Modeling\nIn this section I will show our final models.\n### Efficientnet B1(TF)\n* Trained with TPU using TFRecords provided by the organizer\n* 3 windows as 3 channels by @vaillant\n* Configurations:\n * Batchsize: 768\n * Epochs: 17\n * Scheduler: 9 epochs **1e-3**, 2 epochs **1e-4** and 6 epochs **1e-5**. Three phases, pick the best model at each phase and continue with a new learning rate next phase.\n* Augmentations:\n * rotation, shear, hzoom, wzoom, hshift, wshift by @cdeotte. https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n * mask, basically a cutout, this has **significantly boosted our valid loss**. Implementation shown as below @wuliaokaola.\n\n\n* Results:\n * Around `0.22` image level weighted loss\n\n### ResNet 3D(TF)\n* Randomly take 256 images from each study distributed by position; if the study does not contain 256 images, tile\n* Input shape: [256, 256, 256, 3], output shape: 256(image level) + 9(study level)\n* ResNet 50 Backbone\n* Change Conv2D to Conv3D\n* Keep z dim as input and use it for image level label\n* It can predict image level labels as well but the performance is not as we expected\n* Results:\n\n\n### EfficientNet 3D CNN(TF)\n* Connect 256 **EfficientNet B0**s with a **TimeDistributed** layer \n* use the same identity_block and conv_block like ResNet on the top\n* It can also predict image level labels as well but the performance is not as we expected\n* Results:\n.png?generation=1603765158757751&alt=media)\n\n### CNN + Transformer(Fastai v2 + Pytorch)\n* Resnet34 Backbone\n* 6 Stacked Reformers head \n* Trained with 256x256 images with a **single window** but stacking 5 **consecutive slices as channels**\n* Affine Augmentations\n* Results: around `0.245` on image level and `0.270` on exam level with 5xTTAs\n\n### Blending\n* Blending **EfficientNet 3D CNN** and **ResNet3D** for study level based on OOF:\n.png?generation=1603765267593556&alt=media)\nHere the exam level loss is around `0.19` with weight `[0.71, 0.05, 0.07, 0.38, 0.28, 0.78, 0.71, 0.13, 0.45]` for each label.\n* Blended with \n* Along with our existing image level models(**EfficientNet B1**), we can get around a 0.2 CV\n\n## Hardwares\nYes, this competition is really heavy on hardwares. We also found the problem of IO bottleneck, i.e. GPU cannot run at full power, as shown below. Here are a list of hardwares we primarily use:\n* Kaggle TPU, GCP TPU for Tensorflow development\n* V100(32G) * 2 for Fastai and Pytorch development\n* Rtx Titan(24g) for casual testing\n\n\n## Additional Tips\n* When **blending Tensowflow and Pytorch** models, it's often a good practice to write them as **scripts** as show below.\n\n## Fin\nThanks for reading this solution. Happy kaggling!", "votes": 27 }, { "id": 1061469, "postDate": "2020-10-27T03:04:36.593Z", "content": "thanks for sharing. you guys conducted a lot of experiment in 15 days.
", "rawMarkdown": "thanks for sharing. you guys conducted a lot of experiment in 15 days. ", "votes": 3, "replies": [ { "id": 1061480, "postDate": "2020-10-27T03:10:31.033Z", "content": "😬😬😬doing our best here.
", "rawMarkdown": "😬😬😬doing our best here.", "votes": 1 } ] }, { "id": 1061476, "postDate": "2020-10-27T03:07:09.847Z", "content": "Good job @underwearfitting . Your team was working hard for results. Congrats you and @wuliaokaola on becoming master
", "rawMarkdown": "Good job @underwearfitting . Your team was working hard for results. Congrats you and @wuliaokaola on becoming master", "votes": 4, "replies": [ { "id": 1061481, "postDate": "2020-10-27T03:11:06.153Z", "content": "Thanks! Wish you luck too :D
", "rawMarkdown": "Thanks! Wish you luck too :D", "votes": 2 } ] }, { "id": 1062191, "postDate": "2020-10-27T16:31:04.870Z", "content": "Congratulations @underwearfitting! I enjoyed reading your write-up, hard-working as always :))
", "rawMarkdown": "Congratulations @underwearfitting! I enjoyed reading your write-up, hard-working as always :))", "votes": 1, "replies": [ { "id": 1062235, "postDate": "2020-10-27T17:15:44.263Z", "content": "thanks @amiiiney. We will continue improving :D
", "rawMarkdown": "thanks @amiiiney. We will continue improving :D", "votes": 1 } ] }, { "id": 1063041, "postDate": "2020-10-28T13:03:29.877Z", "content": "Congrats you @underwearfitting and @wuliaokaola on becoming master!!
\nI was also trying to use FastAI but did not get good results and left it midway,
\nwould be great if you shared code in github/kaggle… would love to see the code implementation… Thanks!!
@bacterio our teammate Professor Bacterio is the true fastai master here.
", "rawMarkdown": "@bacterio our teammate Professor Bacterio is the true fastai master here.", "votes": 1 }, { "id": 1063523, "postDate": "2020-10-29T02:51:17.917Z", "content": "Thanks @underwearfitting !!
\nHoping to learn more from Porf @bacterio !!
Hi @kmldas and thanks for your interest! Let me clean up a bit and I'll post a public kernel with the fastai side of our solution in the next few days.
", "rawMarkdown": "Hi @kmldas and thanks for your interest! Let me clean up a bit and I'll post a public kernel with the fastai side of our solution in the next few days.", "votes": 2 }, { "id": 1063785, "postDate": "2020-10-29T10:28:32.213Z", "content": "Thanks so much Porf @bacterio 🖖👍
", "rawMarkdown": "Thanks so much Porf @bacterio 🖖👍" } ] }, { "id": 1062188, "postDate": "2020-10-27T16:29:39.197Z", "content": "Congrats Eric! Happy to see you finally becoming Master, well deserved :)
", "rawMarkdown": "Congrats Eric! Happy to see you finally becoming Master, well deserved :)", "votes": 2, "replies": [ { "id": 1062237, "postDate": "2020-10-27T17:16:18.760Z", "content": "thanks Nico!!! You are GM in the make!!💯
", "rawMarkdown": "thanks Nico!!! You are GM in the make!!💯", "votes": 1 } ] }, { "id": 1064598, "postDate": "2020-10-30T09:52:07.390Z", "content": "Congrats you
", "rawMarkdown": "Congrats you" } ], "comments": [ { "id": 1061469, "author_name": "Tian", "author_url": "", "post_date": "2020-10-27T03:04:36.593000", "content": "thanks for sharing. you guys conducted a lot of experiment in 15 days.
", "votes": 3, "replies": [ { "id": 1061480, "author_name": "sin", "author_url": "", "post_date": "2020-10-27T03:10:31.033000", "content": "😬😬😬doing our best here.
", "votes": 1, "replies": [] } ] }, { "id": 1061476, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-10-27T03:07:09.847000", "content": "Good job @underwearfitting . Your team was working hard for results. Congrats you and @wuliaokaola on becoming master
", "votes": 4, "replies": [ { "id": 1061481, "author_name": "sin", "author_url": "", "post_date": "2020-10-27T03:11:06.153000", "content": "Thanks! Wish you luck too :D
", "votes": 2, "replies": [] } ] }, { "id": 1062191, "author_name": "Amin", "author_url": "", "post_date": "2020-10-27T16:31:04.870000", "content": "Congratulations @underwearfitting! I enjoyed reading your write-up, hard-working as always :))
", "votes": 1, "replies": [ { "id": 1062235, "author_name": "sin", "author_url": "", "post_date": "2020-10-27T17:15:44.263000", "content": "thanks @amiiiney. We will continue improving :D
", "votes": 1, "replies": [] } ] }, { "id": 1063041, "author_name": "Kamal Das", "author_url": "", "post_date": "2020-10-28T13:03:29.877000", "content": "Congrats you @underwearfitting and @wuliaokaola on becoming master!!
\nI was also trying to use FastAI but did not get good results and left it midway,
\nwould be great if you shared code in github/kaggle… would love to see the code implementation… Thanks!!
@bacterio our teammate Professor Bacterio is the true fastai master here.
", "votes": 1, "replies": [] }, { "id": 1063523, "author_name": "Kamal Das", "author_url": "", "post_date": "2020-10-29T02:51:17.917000", "content": "Thanks @underwearfitting !!
\nHoping to learn more from Porf @bacterio !!
Hi @kmldas and thanks for your interest! Let me clean up a bit and I'll post a public kernel with the fastai side of our solution in the next few days.
", "votes": 2, "replies": [] }, { "id": 1063785, "author_name": "Kamal Das", "author_url": "", "post_date": "2020-10-29T10:28:32.213000", "content": "Thanks so much Porf @bacterio 🖖👍
", "votes": 0, "replies": [] } ] }, { "id": 1062188, "author_name": "hrunic", "author_url": "", "post_date": "2020-10-27T16:29:39.197000", "content": "Congrats Eric! Happy to see you finally becoming Master, well deserved :)
", "votes": 2, "replies": [ { "id": 1062237, "author_name": "sin", "author_url": "", "post_date": "2020-10-27T17:16:18.760000", "content": "thanks Nico!!! You are GM in the make!!💯
", "votes": 1, "replies": [] } ] }, { "id": 1064598, "author_name": "huangzhizheng", "author_url": "", "post_date": "2020-10-30T09:52:07.390000", "content": "Congrats you
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1061456": "Thanks to the Kaggle team and organizers for this challenging competition. Huge props to my teammates @wuliaokaola @bacterio, they worked very hard!! We have learned a lot and now we are exhausted both computationally and physically 😃, just like many of you. This solution is developed within 15 days and it's far away from the top winning solutions. But hopefully there's something helpful in this!\n\n## TL; DR\n* **Efficientnet B1** for image level, **Efficientnet 3D CNN** + **Resnet 3D** for study level. **CNN + Transformer** for image and study level.\n\n* Used TFRecords by the organizers and windows function by @vaillant.\n\n* Applied mask to hide left/right side of the image\n\n* Optimized blending weights based on OOF calculation.\n\n## Timeline\n\n* We started the competition about 15 days ago. \n\n* Trained **Efficientnet B0** on image level and blended with the mean predictions by @osciiart, aka the public notebook scored 0.325. 0.4 * mean prediction + 0.6 * Efficient B0 along with the mean prediction for study levels. This gives us LB: `0.292`. \n\n* Developed **Resnet3D** that scored `0.370` LB. We ran a local validation for this model and found out that this model did well on the study level but poorly on the image level. Hence, for study level, we replaced then mean prediction with our Resnet3D and achieved `0.253` LB.\n\n* Improved **Efficientnet B0** with masks(masking one half of the image). And achieved `0.248` LB. \n\n* The amazing public baseline came out and we were frustrated because it scored `0.233` and it made us panic. We tried to inference that and it took forever so we decided to not incorporate that into our pipeline.\n\n* We added @bacterio to our team and blended with our existing results(take the mean) and got `0.226` LB. @bacterio had a **CNN + Transformer** at that time, which is completely disparate from our approach. It calmed us down a bit and we knew that if we continue improving both image level and study level, we will get better.\n\n* We improved by training three **Efficientnet B1**s with similar techniques and @wuliaokaola came out with a new architecture **Efficientnet 3D CNN** and we got `0.212` LB. \n\n* Oh! We only had **two days left**. What should we do? We realized that it's quite unrealistic add in another architecture. So we should work on upgrading existing models. We added TTAs to **Efficientnet 3D CNN** and fine-tuned **Efficientnet B1**, **Resnet 3D**, **Efficientnet 3D CNN** and we got `0.204` LB.\n\n## Modeling\nIn this section I will show our final models.\n### Efficientnet B1(TF)\n* Trained with TPU using TFRecords provided by the organizer\n* 3 windows as 3 channels by @vaillant\n* Configurations:\n * Batchsize: 768\n * Epochs: 17\n * Scheduler: 9 epochs **1e-3**, 2 epochs **1e-4** and 6 epochs **1e-5**. Three phases, pick the best model at each phase and continue with a new learning rate next phase.\n* Augmentations:\n * rotation, shear, hzoom, wzoom, hshift, wshift by @cdeotte. https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n * mask, basically a cutout, this has **significantly boosted our valid loss**. Implementation shown as below @wuliaokaola.\n\n\n* Results:\n * Around `0.22` image level weighted loss\n\n### ResNet 3D(TF)\n* Randomly take 256 images from each study distributed by position; if the study does not contain 256 images, tile\n* Input shape: [256, 256, 256, 3], output shape: 256(image level) + 9(study level)\n* ResNet 50 Backbone\n* Change Conv2D to Conv3D\n* Keep z dim as input and use it for image level label\n* It can predict image level labels as well but the performance is not as we expected\n* Results:\n\n\n### EfficientNet 3D CNN(TF)\n* Connect 256 **EfficientNet B0**s with a **TimeDistributed** layer \n* use the same identity_block and conv_block like ResNet on the top\n* It can also predict image level labels as well but the performance is not as we expected\n* Results:\n.png?generation=1603765158757751&alt=media)\n\n### CNN + Transformer(Fastai v2 + Pytorch)\n* Resnet34 Backbone\n* 6 Stacked Reformers head \n* Trained with 256x256 images with a **single window** but stacking 5 **consecutive slices as channels**\n* Affine Augmentations\n* Results: around `0.245` on image level and `0.270` on exam level with 5xTTAs\n\n### Blending\n* Blending **EfficientNet 3D CNN** and **ResNet3D** for study level based on OOF:\n.png?generation=1603765267593556&alt=media)\nHere the exam level loss is around `0.19` with weight `[0.71, 0.05, 0.07, 0.38, 0.28, 0.78, 0.71, 0.13, 0.45]` for each label.\n* Blended with \n* Along with our existing image level models(**EfficientNet B1**), we can get around a 0.2 CV\n\n## Hardwares\nYes, this competition is really heavy on hardwares. We also found the problem of IO bottleneck, i.e. GPU cannot run at full power, as shown below. Here are a list of hardwares we primarily use:\n* Kaggle TPU, GCP TPU for Tensorflow development\n* V100(32G) * 2 for Fastai and Pytorch development\n* Rtx Titan(24g) for casual testing\n\n\n## Additional Tips\n* When **blending Tensowflow and Pytorch** models, it's often a good practice to write them as **scripts** as show below.\n\n## Fin\nThanks for reading this solution. Happy kaggling!", "1061469": "thanks for sharing. you guys conducted a lot of experiment in 15 days. ", "1061476": "Good job @underwearfitting . Your team was working hard for results. Congrats you and @wuliaokaola on becoming master", "1062191": "Congratulations @underwearfitting! I enjoyed reading your write-up, hard-working as always :))", "1063041": "Congrats you @underwearfitting and @wuliaokaola on becoming master!!\nI was also trying to use FastAI but did not get good results and left it midway,\nwould be great if you shared code in github/kaggle... would love to see the code implementation... Thanks!!\n", "1062188": "Congrats Eric! Happy to see you finally becoming Master, well deserved :)", "1064598": "Congrats you" } }