Thanks to Mayo Clinic and Kaggle for this competition. I enjoyed this competition a lot. Here I would like to describe my solution in short.
\nI think this is the most impactful process in my solution. Because the model weights trained by ImageNet would not be suitable for this task, I used \"other\" data to pretrain networks.
\nI separated \"other\" images into tiles, used the labels of \"Unknown\" or \"Other\", and pretrained ResNet152 models with ImageNet weights for 40 epochs. Then I trained the model to predict CE/LAA labels by using training dataset starting with pretrained weights.
\nIn fact, the single pre-trained ResNet model got 0.65726 in Private Score, which is the best score among my submission.
\n(I did not select the best model as final submission. I selected ensembled models with other no-pretrained methods instead, as described below.)
\nThis is why I think pre-training was the most important process in my solution.
Because the labels of the training data are imbalanced, I used binary cross entropy with balanced class weights.
\nensemble of the following 3 methods
\nThanks for reading.
", "messages": [ { "id": 1975612, "postDate": "2022-10-06T22:37:42.050Z", "content": "Thanks to Mayo Clinic and Kaggle for this competition. I enjoyed this competition a lot. Here I would like to describe my solution in short.
\nI think this is the most impactful process in my solution. Because the model weights trained by ImageNet would not be suitable for this task, I used \"other\" data to pretrain networks.
\nI separated \"other\" images into tiles, used the labels of \"Unknown\" or \"Other\", and pretrained ResNet152 models with ImageNet weights for 40 epochs. Then I trained the model to predict CE/LAA labels by using training dataset starting with pretrained weights.
\nIn fact, the single pre-trained ResNet model got 0.65726 in Private Score, which is the best score among my submission.
\n(I did not select the best model as final submission. I selected ensembled models with other no-pretrained methods instead, as described below.)
\nThis is why I think pre-training was the most important process in my solution.
Because the labels of the training data are imbalanced, I used binary cross entropy with balanced class weights.
\nensemble of the following 3 methods
\nThanks for reading.
", "rawMarkdown": "Thanks to Mayo Clinic and Kaggle for this competition. I enjoyed this competition a lot. Here I would like to describe my solution in short.\n\n# Pre-training\nI think this is the most impactful process in my solution. Because the model weights trained by ImageNet would not be suitable for this task, I used \"other\" data to pretrain networks.\n\nI separated \"other\" images into tiles, used the labels of \"Unknown\" or \"Other\", and pretrained ResNet152 models with ImageNet weights for 40 epochs. Then I trained the model to predict CE/LAA labels by using training dataset starting with pretrained weights.\n\nIn fact, the single pre-trained ResNet model got 0.65726 in Private Score, which is the best score among my submission.\n(I did not select the best model as final submission. I selected ensembled models with other no-pretrained methods instead, as described below.)\nThis is why I think pre-training was the most important process in my solution.\n\n# Data\n\n - tiled into 512x512x3 channels and selected 16 instances for each image\n - saved as tiff format (LZW compression)\n\n# Loss\n Because the labels of the training data are imbalanced, I used binary cross entropy with balanced class weights.\n \n# Models\n ensemble of the following 3 methods\n * Resnet152 with pre-training\n * EfficientNetB0 without pre-training\n * Xception without pre-training\n\nThanks for reading.", "votes": 15 }, { "id": 1977375, "postDate": "2022-10-08T02:00:04.413Z", "content": "Hi @sikeda , Congrats for you solo gold medal. Did your model use MIL head? If so, How did you apply the mil head?
", "rawMarkdown": "Hi @sikeda , Congrats for you solo gold medal. Did your model use MIL head? If so, How did you apply the mil head? ", "replies": [ { "id": 1977660, "postDate": "2022-10-08T06:46:42.680Z", "content": "Hi @forcewithme , thanks for your comment!
\nI did not use MIL head. My model predicts the labels for 16 tiled images independently and takes average of them.
Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 1975966, "postDate": "2022-10-07T05:48:53.883Z", "content": "@sikeda Congrats on the great achievement. I tried to experiemen with resnet models and was able to achieve 0.698 private lb score with just 1 single resnet50 model just on training data. I guess the ensembling in my case would have done the trick. Would love to know more where you think i might have done wrong.
", "rawMarkdown": "@sikeda Congrats on the great achievement. I tried to experiemen with resnet models and was able to achieve 0.698 private lb score with just 1 single resnet50 model just on training data. I guess the ensembling in my case would have done the trick. Would love to know more where you think i might have done wrong. ", "replies": [ { "id": 1976567, "postDate": "2022-10-07T12:34:15.417Z", "content": "Thanks for your comment!
\nI have not submitted resnet50 models for this competition, but I got 0.66579 for single ResNet152 without pre-training and 0.67792 for xception without pre-training.
\nI think your resnet50 model would have scored around 0.666~0.680 or so if the model was trained appropriately.
\nThus, I can imagine that there is something to improve in pre-processing or training process.
For example, did you use balanced cross entropy to train the model?
\nWhile the training dataset is imbalanced (CE:LAA = 73:27), the test dataset is maybe balanced (CE:LAA = roughly 1:1).
\nIf you use simple cross entropy and do not care about the imbalance, the model output will be biased to around 0.73 of CE probability.
\nOf course it is OK when the test dataset is similarly imbalanced, but in this competition it is not (maybe).
Congrats! Thank you for sharing your work!
", "rawMarkdown": "Congrats! Thank you for sharing your work!", "replies": [ { "id": 1976569, "postDate": "2022-10-07T12:34:41.047Z", "content": "Thanks a lot for your comment!
", "rawMarkdown": "Thanks a lot for your comment!" } ] }, { "id": 1975832, "postDate": "2022-10-07T03:43:09.907Z", "content": "@sikeda congrats on your results, and unlucky that you missed out a potentially winning solution :)
\nYou mention a very interesting pretraining process, could you elaborate a little bit on how you used 'Unknown' and 'Other' for pretraining? Specifically what target labels did you use for this training portion, and how do you think this helped?
", "rawMarkdown": "@sikeda congrats on your results, and unlucky that you missed out a potentially winning solution :)\n\nYou mention a very interesting pretraining process, could you elaborate a little bit on how you used 'Unknown' and 'Other' for pretraining? Specifically what target labels did you use for this training portion, and how do you think this helped? ", "replies": [ { "id": 1976597, "postDate": "2022-10-07T12:55:40.327Z", "content": "Thank you and congrats on your result!
\nFor pre-training, I just used the label as is ('Unknown':0, 'Other':1).
\nI know the labels are different from the competition task, but similar to some extent.
\nThus I expected that the network would learn to retrieve some useful information from the competition images.
Thanks, this is such an interesting approach. I tried to do this together while training, but I guess by pretraining you can control the two training phases separately and that could be helpful. Great job.
", "rawMarkdown": "Thanks, this is such an interesting approach. I tried to do this together while training, but I guess by pretraining you can control the two training phases separately and that could be helpful. Great job." } ] } ], "comments": [ { "id": 1977375, "author_name": "ForcewithMe", "author_url": "", "post_date": "2022-10-08T02:00:04.413000", "content": "Hi @sikeda , Congrats for you solo gold medal. Did your model use MIL head? If so, How did you apply the mil head?
", "votes": 0, "replies": [ { "id": 1977660, "author_name": "miyasaki", "author_url": "", "post_date": "2022-10-08T06:46:42.680000", "content": "Hi @forcewithme , thanks for your comment!
\nI did not use MIL head. My model predicts the labels for 16 tiled images independently and takes average of them.
Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 1975966, "author_name": "Mrinal Tyagi", "author_url": "", "post_date": "2022-10-07T05:48:53.883000", "content": "@sikeda Congrats on the great achievement. I tried to experiemen with resnet models and was able to achieve 0.698 private lb score with just 1 single resnet50 model just on training data. I guess the ensembling in my case would have done the trick. Would love to know more where you think i might have done wrong.
", "votes": 0, "replies": [ { "id": 1976567, "author_name": "miyasaki", "author_url": "", "post_date": "2022-10-07T12:34:15.417000", "content": "Thanks for your comment!
\nI have not submitted resnet50 models for this competition, but I got 0.66579 for single ResNet152 without pre-training and 0.67792 for xception without pre-training.
\nI think your resnet50 model would have scored around 0.666~0.680 or so if the model was trained appropriately.
\nThus, I can imagine that there is something to improve in pre-processing or training process.
For example, did you use balanced cross entropy to train the model?
\nWhile the training dataset is imbalanced (CE:LAA = 73:27), the test dataset is maybe balanced (CE:LAA = roughly 1:1).
\nIf you use simple cross entropy and do not care about the imbalance, the model output will be biased to around 0.73 of CE probability.
\nOf course it is OK when the test dataset is similarly imbalanced, but in this competition it is not (maybe).
Congrats! Thank you for sharing your work!
", "votes": 0, "replies": [ { "id": 1976569, "author_name": "miyasaki", "author_url": "", "post_date": "2022-10-07T12:34:41.047000", "content": "Thanks a lot for your comment!
", "votes": 0, "replies": [] } ] }, { "id": 1975832, "author_name": "tdiceman", "author_url": "", "post_date": "2022-10-07T03:43:09.907000", "content": "@sikeda congrats on your results, and unlucky that you missed out a potentially winning solution :)
\nYou mention a very interesting pretraining process, could you elaborate a little bit on how you used 'Unknown' and 'Other' for pretraining? Specifically what target labels did you use for this training portion, and how do you think this helped?
", "votes": 0, "replies": [ { "id": 1976597, "author_name": "miyasaki", "author_url": "", "post_date": "2022-10-07T12:55:40.327000", "content": "Thank you and congrats on your result!
\nFor pre-training, I just used the label as is ('Unknown':0, 'Other':1).
\nI know the labels are different from the competition task, but similar to some extent.
\nThus I expected that the network would learn to retrieve some useful information from the competition images.
Thanks, this is such an interesting approach. I tried to do this together while training, but I guess by pretraining you can control the two training phases separately and that could be helpful. Great job.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1975612": "Thanks to Mayo Clinic and Kaggle for this competition. I enjoyed this competition a lot. Here I would like to describe my solution in short.\n\n# Pre-training\nI think this is the most impactful process in my solution. Because the model weights trained by ImageNet would not be suitable for this task, I used \"other\" data to pretrain networks.\n\nI separated \"other\" images into tiles, used the labels of \"Unknown\" or \"Other\", and pretrained ResNet152 models with ImageNet weights for 40 epochs. Then I trained the model to predict CE/LAA labels by using training dataset starting with pretrained weights.\n\nIn fact, the single pre-trained ResNet model got 0.65726 in Private Score, which is the best score among my submission.\n(I did not select the best model as final submission. I selected ensembled models with other no-pretrained methods instead, as described below.)\nThis is why I think pre-training was the most important process in my solution.\n\n# Data\n\n - tiled into 512x512x3 channels and selected 16 instances for each image\n - saved as tiff format (LZW compression)\n\n# Loss\n Because the labels of the training data are imbalanced, I used binary cross entropy with balanced class weights.\n \n# Models\n ensemble of the following 3 methods\n * Resnet152 with pre-training\n * EfficientNetB0 without pre-training\n * Xception without pre-training\n\nThanks for reading.", "1977375": "Hi @sikeda , Congrats for you solo gold medal. Did your model use MIL head? If so, How did you apply the mil head? ", "1975966": "@sikeda Congrats on the great achievement. I tried to experiemen with resnet models and was able to achieve 0.698 private lb score with just 1 single resnet50 model just on training data. I guess the ensembling in my case would have done the trick. Would love to know more where you think i might have done wrong. ", "1975906": "Congrats! Thank you for sharing your work!", "1975832": "@sikeda congrats on your results, and unlucky that you missed out a potentially winning solution :)\n\nYou mention a very interesting pretraining process, could you elaborate a little bit on how you used 'Unknown' and 'Other' for pretraining? Specifically what target labels did you use for this training portion, and how do you think this helped? " } }