Preprocessing
\n1) I annotated part of the NIH data and trained a lung localizer for more efficient usage of gpu memory. The drawback is that diseases outside lung area is guaranteed missing.
\n2) I treated each image-annotation pair as independent data. So both the training and validation data were tripled.
Modeling
\nSimilar to the majority, I used object detection for disease localization, and binary classifiers for predicting \"no finding\" as well as removing false positive bboxes on healthy images.
For detection, I trained efficientdet-d5 on positive images and validated on the full validation set including healthy images. The scores were 5 fold average CV: 0.36, public LB: 0.191, private LB: 0.219. I used WBF (https://github.com/ZFTurbo/Weighted-Boxes-Fusion) for ensembling. The image input size was 1152x1152.
\nFor binary classifier training, I added an additional multi-label head which gave slightly better performance. The CV was about 0.994 AUC with a blend of efficientnet-b7 and resnet200d. The image input size was 640x640.
\nTo fuse the detection and classification models, I first multiplied the confidence scores of every bbox by Prob(unhealthy)^0.2, then I added [14, Prob(healthy), 0, 0, 1, 1] to all the predictions. With the classifiers, CV improved to 0.47, public improved to 0.266, private improved to 0.297.
\nWhat didn't work
\n1) Yolov5 with default parameters performed much worse than efficientdet. Due to the time limit, I gave up yolov5 easily. I should have spent more time on it.
\n2) I tried different kinds of pseudo-labeling on most of the external data but nothing worked.
Preprocessing
\n1) I annotated part of the NIH data and trained a lung localizer for more efficient usage of gpu memory. The drawback is that diseases outside lung area is guaranteed missing.
\n2) I treated each image-annotation pair as independent data. So both the training and validation data were tripled.
Modeling
\nSimilar to the majority, I used object detection for disease localization, and binary classifiers for predicting \"no finding\" as well as removing false positive bboxes on healthy images.
For detection, I trained efficientdet-d5 on positive images and validated on the full validation set including healthy images. The scores were 5 fold average CV: 0.36, public LB: 0.191, private LB: 0.219. I used WBF (https://github.com/ZFTurbo/Weighted-Boxes-Fusion) for ensembling. The image input size was 1152x1152.
\nFor binary classifier training, I added an additional multi-label head which gave slightly better performance. The CV was about 0.994 AUC with a blend of efficientnet-b7 and resnet200d. The image input size was 640x640.
\nTo fuse the detection and classification models, I first multiplied the confidence scores of every bbox by Prob(unhealthy)^0.2, then I added [14, Prob(healthy), 0, 0, 1, 1] to all the predictions. With the classifiers, CV improved to 0.47, public improved to 0.266, private improved to 0.297.
\nWhat didn't work
\n1) Yolov5 with default parameters performed much worse than efficientdet. Due to the time limit, I gave up yolov5 easily. I should have spent more time on it.
\n2) I tried different kinds of pseudo-labeling on most of the external data but nothing worked.
Congrats! I also tried pseudo-labeling with both competition test data and NIH datasets but didn't work on my side either. And yes, you should have used yolov5 predictions in your ensembles at least..
", "rawMarkdown": "Congrats! I also tried pseudo-labeling with both competition test data and NIH datasets but didn't work on my side either. And yes, you should have used yolov5 predictions in your ensembles at least..", "votes": 1, "replies": [ { "id": 1258482, "postDate": "2021-03-31T16:46:38.543Z", "content": "It gave only tiny boost to us, our efficientdets were also better than yolo.
", "rawMarkdown": "It gave only tiny boost to us, our efficientdets were also better than yolo." }, { "id": 1258483, "postDate": "2021-03-31T16:48:51.677Z", "content": "How did you ensemble yolo preds? It seems other top solutions also include yolo predictions.
", "rawMarkdown": "How did you ensemble yolo preds? It seems other top solutions also include yolo predictions. " }, { "id": 1258506, "postDate": "2021-03-31T17:14:17.063Z", "content": "It always depends on how strong and diverse your other models are. We blended with WBF.
", "rawMarkdown": "It always depends on how strong and diverse your other models are. We blended with WBF." }, { "id": 1258545, "postDate": "2021-03-31T17:43:59.600Z", "content": "You are right, it depends on what you are ensembling with. I personally found out that detectron2 and yolo was blending well. I was trying to say it was definitely worth to try YOLO preds in an ensemble before giving up on them.
", "rawMarkdown": "You are right, it depends on what you are ensembling with. I personally found out that detectron2 and yolo was blending well. I was trying to say it was definitely worth to try YOLO preds in an ensemble before giving up on them.", "votes": 1 } ] }, { "id": 1258464, "postDate": "2021-03-31T16:18:28.383Z", "content": "@wowfattie Congrats. You're like a Ninja in LB, can't be seen in public lb but tops on the private one 💥
", "rawMarkdown": "@wowfattie Congrats. You're like a Ninja in LB, can't be seen in public lb but tops on the private one 💥", "votes": 2 }, { "id": 1320955, "postDate": "2021-05-24T12:46:56.603Z", "content": "Can you share with us the GitHub link from where you have implemented the Effdet code?
", "rawMarkdown": "Can you share with us the GitHub link from where you have implemented the Effdet code?", "replies": [ { "id": 1320957, "postDate": "2021-05-24T12:49:52.407Z", "content": "And can you tell me the intuition behind your postprocessing that is I first multiplied the confidence scores of every bbox by Prob(unhealthy)^0.2, then I added [14, Prob(healthy), 0, 0, 1, 1] to all the predictions.
Congratulations @wowfattie, a sort of kangaroo in the LB
", "rawMarkdown": "Congratulations @wowfattie, a sort of kangaroo in the LB" }, { "id": 1259039, "postDate": "2021-04-01T05:24:50.877Z", "content": "Congratz and thank you for the nice summary.
\nI have a single question regarding the training of the EfficientDet D5. How long did you train it and on what type of GPU ?
", "rawMarkdown": "Congratz and thank you for the nice summary.\n\nI have a single question regarding the training of the EfficientDet D5. How long did you train it and on what type of GPU ? " }, { "id": 1258804, "postDate": "2021-03-31T22:43:09.070Z", "content": "Congratulations Guanshuo. I'm impressed that you joined the competition late and did so well. That's a very accurate Yolo model. Our single model Yolo has private LB 290. Our ensemble of Yolo, VFNet, and EffDet achieves private LB 296. If you had time to build a VFNet and EffDet and ensemble 3 models, your score would be so high! Well done!
", "rawMarkdown": "Congratulations Guanshuo. I'm impressed that you joined the competition late and did so well. That's a very accurate Yolo model. Our single model Yolo has private LB 290. Our ensemble of Yolo, VFNet, and EffDet achieves private LB 296. If you had time to build a VFNet and EffDet and ensemble 3 models, your score would be so high! Well done!", "replies": [ { "id": 1258810, "postDate": "2021-03-31T22:50:04.743Z", "content": "Thanks, Chris. Yes, I realized after reading other winning solutions that ensembling different models is more effective than ensembling same models of different CV fold. BTW, my best model is efficientdet not yolo.
", "rawMarkdown": "Thanks, Chris. Yes, I realized after reading other winning solutions that ensembling different models is more effective than ensembling same models of different CV fold. BTW, my best model is efficientdet not yolo.", "votes": 2 }, { "id": 1258822, "postDate": "2021-03-31T23:14:01.543Z", "content": "Oh right, EfficientDet D5, sorry, I read your post earlier and was impressed that you only used a single model. But i remembered the model type wrong.
\nFor us YoloV5 > VFNet > EffDet D4 with private scores LB 290 > LB 289 > LB 282. But we spent the least amount of time tuning EffDet. We trained and added that model on the last night.
", "rawMarkdown": "Oh right, EfficientDet D5, sorry, I read your post earlier and was impressed that you only used a single model. But i remembered the model type wrong. \n\nFor us YoloV5 > VFNet > EffDet D4 with private scores LB 290 > LB 289 > LB 282. But we spent the least amount of time tuning EffDet. We trained and added that model on the last night." }, { "id": 1259059, "postDate": "2021-04-01T05:50:11.507Z", "content": "@cdeotte Hi, I have a question. Does your Yolov5 ensemble method achieve 0.290? How many folds did you use, and what is the training size, you apply WBF or you use average weight functions of yolo5? and what is the average number of boxes in each predicted image?
", "rawMarkdown": "@cdeotte Hi, I have a question. Does your Yolov5 ensemble method achieve 0.290? How many folds did you use, and what is the training size, you apply WBF or you use average weight functions of yolo5? and what is the average number of boxes in each predicted image?" }, { "id": 1259689, "postDate": "2021-04-01T15:33:01.220Z", "content": "@hoangduyloc Yes. Our single Yolo model scores private LB 290 and achieves 12th place Gold. It is 10 stratified folds. Each fold is processed with NMS IOU 0.4 and threshold 0.001. Then the 10 folds are combined with WBF IOU 0.4 with conf_type = 'max'. Finally we apply our post process using our classifier model.
Note that 4th place team here, achieved their final position with a single YoloV5 model too.
\nAnd note that Guanshuo achieved his 6th place with a single EfficientDet D5. It shows that single models can win Gold in this competition. I like that.
", "rawMarkdown": "@hoangduyloc Yes. Our single Yolo model scores private LB 290 and achieves 12th place Gold. It is 10 stratified folds. Each fold is processed with NMS IOU 0.4 and threshold 0.001. Then the 10 folds are combined with WBF IOU 0.4 with `conf_type = 'max'`. Finally we apply our post process using our classifier model.\n\nNote that 4th place team [here][1], achieved their final position with a single YoloV5 model too. \n\nAnd note that Guanshuo achieved his 6th place with a single EfficientDet D5. It shows that single models can win Gold in this competition. I like that.\n\n[1]: https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229786", "votes": 1 }, { "id": 1260270, "postDate": "2021-04-02T01:42:53.427Z", "content": "@cdeotte Tthank you so much, this is my first competition on Kaggle ever. At first, I find SATO method on Coco and see that Scaled-yolov4 is the best,, I also apply it to this competition, my single fold of Scaledyolov4 also high at 0.5/0.238/0.256 for CV/LB/PV but unfortunately, I'm doing some wrongs in the ensemble steps which lead me to overfit on LB 0.274 but private reduce 0.231. I think if you guys use Scaledyolov4 the score will be more higher!!!
", "rawMarkdown": "@cdeotte Tthank you so much, this is my first competition on Kaggle ever. At first, I find SATO method on Coco and see that Scaled-yolov4 is the best,, I also apply it to this competition, my single fold of Scaledyolov4 also high at 0.5/0.238/0.256 for CV/LB/PV but unfortunately, I'm doing some wrongs in the ensemble steps which lead me to overfit on LB 0.274 but private reduce 0.231. I think if you guys use Scaledyolov4 the score will be more higher!!!" }, { "id": 1264162, "postDate": "2021-04-06T00:08:45.530Z", "content": "@cdeotte
\nGuanshuo mentioned that default yolov5 parameters does not work very well, but for me the default yolov5 performed much better than effdetd5.
\nDid you change any training parameters on yolo? I tried to fiddle image size and mixup ratio, but the CV was quite constant.
2) I treated each image-annotation pair as independent data. So both the training and validation data were tripled.
\nI couldnt get this point
It simply means I did not do any preprocessing on the labels. Note that most of the winners combined labels from three annotators (using wbf for example) before training. I did not do that.
", "rawMarkdown": "It simply means I did not do any preprocessing on the labels. Note that most of the winners combined labels from three annotators (using wbf for example) before training. I did not do that.", "votes": 2 }, { "id": 1259172, "postDate": "2021-04-01T07:36:15.197Z", "content": "Very interesting approach @wowfattie :D
\nHave you done any ablation study on tripled-and-separated annotations vs WBF annotations?
No I didn't have time to try other kinds of preprocessing
", "rawMarkdown": "No I didn't have time to try other kinds of preprocessing\n" } ] }, { "id": 1259032, "postDate": "2021-04-01T05:16:26.957Z", "content": "Thanks for sharing information.
", "rawMarkdown": "Thanks for sharing information." } ], "comments": [ { "id": 1258478, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2021-03-31T16:36:38.343000", "content": "Congrats! I also tried pseudo-labeling with both competition test data and NIH datasets but didn't work on my side either. And yes, you should have used yolov5 predictions in your ensembles at least..
", "votes": 1, "replies": [ { "id": 1258482, "author_name": "Psi", "author_url": "", "post_date": "2021-03-31T16:46:38.543000", "content": "It gave only tiny boost to us, our efficientdets were also better than yolo.
", "votes": 0, "replies": [] }, { "id": 1258483, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2021-03-31T16:48:51.677000", "content": "How did you ensemble yolo preds? It seems other top solutions also include yolo predictions.
", "votes": 0, "replies": [] }, { "id": 1258506, "author_name": "Psi", "author_url": "", "post_date": "2021-03-31T17:14:17.063000", "content": "It always depends on how strong and diverse your other models are. We blended with WBF.
", "votes": 0, "replies": [] }, { "id": 1258545, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2021-03-31T17:43:59.600000", "content": "You are right, it depends on what you are ensembling with. I personally found out that detectron2 and yolo was blending well. I was trying to say it was definitely worth to try YOLO preds in an ensemble before giving up on them.
", "votes": 1, "replies": [] } ] }, { "id": 1258464, "author_name": "Awsaf", "author_url": "", "post_date": "2021-03-31T16:18:28.383000", "content": "@wowfattie Congrats. You're like a Ninja in LB, can't be seen in public lb but tops on the private one 💥
", "votes": 2, "replies": [] }, { "id": 1320955, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-05-24T12:46:56.603000", "content": "Can you share with us the GitHub link from where you have implemented the Effdet code?
", "votes": 0, "replies": [ { "id": 1320957, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-05-24T12:49:52.407000", "content": "And can you tell me the intuition behind your postprocessing that is I first multiplied the confidence scores of every bbox by Prob(unhealthy)^0.2, then I added [14, Prob(healthy), 0, 0, 1, 1] to all the predictions.
Congratulations @wowfattie, a sort of kangaroo in the LB
", "votes": 0, "replies": [] }, { "id": 1259039, "author_name": "wawa.", "author_url": "", "post_date": "2021-04-01T05:24:50.877000", "content": "Congratz and thank you for the nice summary.
\nI have a single question regarding the training of the EfficientDet D5. How long did you train it and on what type of GPU ?
", "votes": 0, "replies": [] }, { "id": 1258804, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T22:43:09.070000", "content": "Congratulations Guanshuo. I'm impressed that you joined the competition late and did so well. That's a very accurate Yolo model. Our single model Yolo has private LB 290. Our ensemble of Yolo, VFNet, and EffDet achieves private LB 296. If you had time to build a VFNet and EffDet and ensemble 3 models, your score would be so high! Well done!
", "votes": 0, "replies": [ { "id": 1258810, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2021-03-31T22:50:04.743000", "content": "Thanks, Chris. Yes, I realized after reading other winning solutions that ensembling different models is more effective than ensembling same models of different CV fold. BTW, my best model is efficientdet not yolo.
", "votes": 2, "replies": [] }, { "id": 1258822, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T23:14:01.543000", "content": "Oh right, EfficientDet D5, sorry, I read your post earlier and was impressed that you only used a single model. But i remembered the model type wrong.
\nFor us YoloV5 > VFNet > EffDet D4 with private scores LB 290 > LB 289 > LB 282. But we spent the least amount of time tuning EffDet. We trained and added that model on the last night.
", "votes": 0, "replies": [] }, { "id": 1259059, "author_name": "David", "author_url": "", "post_date": "2021-04-01T05:50:11.507000", "content": "@cdeotte Hi, I have a question. Does your Yolov5 ensemble method achieve 0.290? How many folds did you use, and what is the training size, you apply WBF or you use average weight functions of yolo5? and what is the average number of boxes in each predicted image?
", "votes": 0, "replies": [] }, { "id": 1259689, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-04-01T15:33:01.220000", "content": "@hoangduyloc Yes. Our single Yolo model scores private LB 290 and achieves 12th place Gold. It is 10 stratified folds. Each fold is processed with NMS IOU 0.4 and threshold 0.001. Then the 10 folds are combined with WBF IOU 0.4 with conf_type = 'max'. Finally we apply our post process using our classifier model.
Note that 4th place team here, achieved their final position with a single YoloV5 model too.
\nAnd note that Guanshuo achieved his 6th place with a single EfficientDet D5. It shows that single models can win Gold in this competition. I like that.
", "votes": 1, "replies": [] }, { "id": 1260270, "author_name": "David", "author_url": "", "post_date": "2021-04-02T01:42:53.427000", "content": "@cdeotte Tthank you so much, this is my first competition on Kaggle ever. At first, I find SATO method on Coco and see that Scaled-yolov4 is the best,, I also apply it to this competition, my single fold of Scaledyolov4 also high at 0.5/0.238/0.256 for CV/LB/PV but unfortunately, I'm doing some wrongs in the ensemble steps which lead me to overfit on LB 0.274 but private reduce 0.231. I think if you guys use Scaledyolov4 the score will be more higher!!!
", "votes": 0, "replies": [] }, { "id": 1264162, "author_name": "arutema47", "author_url": "", "post_date": "2021-04-06T00:08:45.530000", "content": "@cdeotte
\nGuanshuo mentioned that default yolov5 parameters does not work very well, but for me the default yolov5 performed much better than effdetd5.
\nDid you change any training parameters on yolo? I tried to fiddle image size and mixup ratio, but the CV was quite constant.
2) I treated each image-annotation pair as independent data. So both the training and validation data were tripled.
\nI couldnt get this point
It simply means I did not do any preprocessing on the labels. Note that most of the winners combined labels from three annotators (using wbf for example) before training. I did not do that.
", "votes": 2, "replies": [] }, { "id": 1259172, "author_name": "DatNT", "author_url": "", "post_date": "2021-04-01T07:36:15.197000", "content": "Very interesting approach @wowfattie :D
\nHave you done any ablation study on tripled-and-separated annotations vs WBF annotations?
No I didn't have time to try other kinds of preprocessing
", "votes": 0, "replies": [] } ] }, { "id": 1259032, "author_name": "youngjoogist", "author_url": "", "post_date": "2021-04-01T05:16:26.957000", "content": "Thanks for sharing information.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1258458": "**Preprocessing**\n1) I annotated part of the NIH data and trained a lung localizer for more efficient usage of gpu memory. The drawback is that diseases outside lung area is guaranteed missing.\n2) I treated each image-annotation pair as independent data. So both the training and validation data were tripled.\n\n**Modeling**\nSimilar to the majority, I used object detection for disease localization, and binary classifiers for predicting \"no finding\" as well as removing false positive bboxes on healthy images.\n\nFor detection, I trained efficientdet-d5 on positive images and validated on the full validation set including healthy images. The scores were 5 fold average CV: 0.36, public LB: 0.191, private LB: 0.219. I used WBF (https://github.com/ZFTurbo/Weighted-Boxes-Fusion) for ensembling. The image input size was 1152x1152.\n\nFor binary classifier training, I added an additional multi-label head which gave slightly better performance. The CV was about 0.994 AUC with a blend of efficientnet-b7 and resnet200d. The image input size was 640x640.\n\nTo fuse the detection and classification models, I first multiplied the confidence scores of every bbox by Prob(unhealthy)^0.2, then I added [14, Prob(healthy), 0, 0, 1, 1] to all the predictions. With the classifiers, CV improved to 0.47, public improved to 0.266, private improved to 0.297.\n\n**What didn't work**\n1) Yolov5 with default parameters performed much worse than efficientdet. Due to the time limit, I gave up yolov5 easily. I should have spent more time on it. \n2) I tried different kinds of pseudo-labeling on most of the external data but nothing worked.\n", "1258478": "Congrats! I also tried pseudo-labeling with both competition test data and NIH datasets but didn't work on my side either. And yes, you should have used yolov5 predictions in your ensembles at least..", "1258464": "@wowfattie Congrats. You're like a Ninja in LB, can't be seen in public lb but tops on the private one 💥", "1320955": "Can you share with us the GitHub link from where you have implemented the Effdet code?", "1259887": "Congratulations @wowfattie, a sort of kangaroo in the LB", "1259039": "Congratz and thank you for the nice summary.\n\nI have a single question regarding the training of the EfficientDet D5. How long did you train it and on what type of GPU ? ", "1258804": "Congratulations Guanshuo. I'm impressed that you joined the competition late and did so well. That's a very accurate Yolo model. Our single model Yolo has private LB 290. Our ensemble of Yolo, VFNet, and EffDet achieves private LB 296. If you had time to build a VFNet and EffDet and ensemble 3 models, your score would be so high! Well done!", "1258734": "`2) I treated each image-annotation pair as independent data. So both the training and validation data were tripled.`\nI couldnt get this point", "1259032": "Thanks for sharing information." } }