At first I started to try different OD models for this dataset. I tried: EffDet (B5), CenterNet (HourGlass), Yolo_v5, Retinanet (RsNet50, ResNet101, ResNet152), Faster RCNN
\nMain initial tricks:
\nBest models were the following:
\nEffDet – same code as Wheat competition, but works pretty poor on this dataset. With Faster-RCNN I think I have some problem with code, because it has the very bad score.
\nUsing WBF ensemble of these models + 2 public models (Detectron2 and Yolo_v5):
\nI was able to achieve 0.300 on Public LB (0.289 Private LB). Then I merge with my teammates. They had more mmdetection models as well as Yolo_v5, but together we were able to got only 0.301.
\nWhile we were stuck we decided to get individual score for each class and compare with our OOF validation. For 3 days we got the following table:
\nClass 0: 0.014 (mAP LB: 0.210 Valid: 0.900)\nClass 1: 0.009 (mAP LB: 0.135 Valid: 0.335)\nClass 2: 0.017 (mAP LB: 0.255 Valid: 0.278)\nClass 3: 0.046 (mAP LB: 0.690 Valid: 0.915)\nClass 4: 0.019 (mAP LB: 0.285 Valid: 0.438)\nClass 5: 0.013 (mAP LB: 0.195 Valid: 0.376)\nClass 6: 0.010 (mAP LB: 0.150 Valid: 0.465)\nClass 7: 0.003 (mAP LB: 0.045 Valid: 0.400)\nClass 8: 0.011 (mAP LB: 0.165 Valid: 0.372)\nClass 9: 0.004 (mAP LB: 0.060 Valid: 0.162)\nClass 10: 0.027 (mAP LB: 0.405 Valid: 0.548)\nClass 11: 0.012 (mAP LB: 0.180 Valid: 0.343)\nClass 12: 0.037 (mAP LB: 0.555 Valid: 0.388)\nClass 13: 0.009 (mAP LB: 0.135 Valid: 0.432)\nClass 14: 0.063 (mAP LB: 0.945 Valid: 0.772)\nSum: 0.294\nAs it can be seen LB mAP the very different from local mAP. We started from investigation of Class 0 difference (it was most suspicious). We have near perfect prediction on validation while LB score is so small. The first thing we tried is to calculate validation mAP based on Radiologist ID.
\nClass ID: 0 Validation file: ensemble_iou_0.4._mAP_0.454_extracted_classes_0.csv\nRadiologist: R8 mAP: 0.991058\nRadiologist: R9 mAP: 0.954593\nRadiologist: R10 mAP: 0.971534\nRadiologist: R11 mAP: 0.379283\nRadiologist: R12 mAP: 0.019819\nRadiologist: R13 mAP: 0.511284\nRadiologist: R14 mAP: 0.527909\nRadiologist: R15 mAP: 0.279870\nRadiologist: R16 mAP: 0.506167\nRadiologist: R17 mAP: 0.447092\nInteresting, we have perfect match for R8, R9 and R10, but very poor on other radiologists. Bad thing is that there are too small entries by other radiologists in train:
\nR11 - Class: 0 Entries: 30 AP: 0.379283\nR12 - Class: 0 Entries: 12 AP: 0.019819\nR13 - Class: 0 Entries: 32 AP: 0.511284\nR14 - Class: 0 Entries: 74 AP: 0.527909\nR15 - Class: 0 Entries: 25 AP: 0.279870\nR16 - Class: 0 Entries: 23 AP: 0.506167\nR17 - Class: 0 Entries: 10 AP: 0.447092\nOne more thing that R8, R9 and R10 never markup with any other radiologists. And what if test set marked up by some other radiologists (may be even by R13, R14 etc) and making boxes too similar to these 3 we overfit to train? And what if with better models we went further from test mark up?
\nI checked by eyes images of class 0 in train marked up by R8, R9 and R10 and by rare radiologists to check the difference. I was lucky to see that other radiologists often use much larger boxes comparing to R8, R9 and R10. So I made a small trick by hand:
\nx1 -= 100 y2 += 100 for all boxes with class 0
\nAnd checked local validation, which improved a lot for other radiologists, while R8, R9 and R10 became worse:
Radiologist: R10 Class: 0 Entries: 2349 AP: 0.559458\nRadiologist: R11 Class: 0 Entries: 30 AP: 0.559355\nRadiologist: R12 Class: 0 Entries: 12 AP: 0.351739\nRadiologist: R13 Class: 0 Entries: 32 AP: 0.483730\nRadiologist: R14 Class: 0 Entries: 74 AP: 0.623019\nRadiologist: R15 Class: 0 Entries: 25 AP: 0.341545\nRadiologist: R16 Class: 0 Entries: 23 AP: 0.506167\nRadiologist: R17 Class: 0 Entries: 10 AP: 0.452887\nI made the same trick on our 0.301 submission and got 0.309 on LB. This showed us the right direction. We finetuned our models using only rare radiologists. And it improved LB score for all of models:
\nRetinaNet (ResNet101): 0.246 -> 0.264
\nRetinaNet (ResNet152): 0.222 -> 0.237
\nYolo_v5: 0.251 -> 0.271
\nEnsemble for all models including original moved us to Public LB > 0.330
At this point it became hard to add new models to ensemble without proper validation. So we decided to switch to single class improvements. It was much easier to find proper stop epochs for this case. Also training procedure became more complicated. We added NIH dataset boxes for classes where it was possible. We prepared NIH dataset in format of VinBigData Chest X-ray competition. We added it as separate radiologist R99. We also prepeared dataset for Pneumothorax, based on SIIM competition, but it didn't help.
\nWe used only one RetinaNet-R101 model for finetuning. We started from best previous weights obtained for multiclass case. So finetuning was pretty fast. We changed sampling in following way:
\nIn parallel we fine-tuned some models on very high resolution (2048px). Some classes from these models catch small boxes better than previous models. It improved our local validation.
\nThis approaches together allowed us to improve even more: >0.36 on public
Actually we decided to create 2 submissions: best on validation and best on LB. We made first submission pretty early. So at the late stage of competition we mostly work on LB-overfitting. That was a little bit risky, but after private LB was revealed we found out it was right thing to do. )
\nYou can find code for out solution on github:
\nhttps://github.com/ZFTurbo/2nd-place-solution-for-VinBigData-Chest-X-ray-Abnormalities-Detection
At first I started to try different OD models for this dataset. I tried: EffDet (B5), CenterNet (HourGlass), Yolo_v5, Retinanet (RsNet50, ResNet101, ResNet152), Faster RCNN
\nMain initial tricks:
\nBest models were the following:
\nEffDet – same code as Wheat competition, but works pretty poor on this dataset. With Faster-RCNN I think I have some problem with code, because it has the very bad score.
\nUsing WBF ensemble of these models + 2 public models (Detectron2 and Yolo_v5):
\nI was able to achieve 0.300 on Public LB (0.289 Private LB). Then I merge with my teammates. They had more mmdetection models as well as Yolo_v5, but together we were able to got only 0.301.
\nWhile we were stuck we decided to get individual score for each class and compare with our OOF validation. For 3 days we got the following table:
\nClass 0: 0.014 (mAP LB: 0.210 Valid: 0.900)\nClass 1: 0.009 (mAP LB: 0.135 Valid: 0.335)\nClass 2: 0.017 (mAP LB: 0.255 Valid: 0.278)\nClass 3: 0.046 (mAP LB: 0.690 Valid: 0.915)\nClass 4: 0.019 (mAP LB: 0.285 Valid: 0.438)\nClass 5: 0.013 (mAP LB: 0.195 Valid: 0.376)\nClass 6: 0.010 (mAP LB: 0.150 Valid: 0.465)\nClass 7: 0.003 (mAP LB: 0.045 Valid: 0.400)\nClass 8: 0.011 (mAP LB: 0.165 Valid: 0.372)\nClass 9: 0.004 (mAP LB: 0.060 Valid: 0.162)\nClass 10: 0.027 (mAP LB: 0.405 Valid: 0.548)\nClass 11: 0.012 (mAP LB: 0.180 Valid: 0.343)\nClass 12: 0.037 (mAP LB: 0.555 Valid: 0.388)\nClass 13: 0.009 (mAP LB: 0.135 Valid: 0.432)\nClass 14: 0.063 (mAP LB: 0.945 Valid: 0.772)\nSum: 0.294\nAs it can be seen LB mAP the very different from local mAP. We started from investigation of Class 0 difference (it was most suspicious). We have near perfect prediction on validation while LB score is so small. The first thing we tried is to calculate validation mAP based on Radiologist ID.
\nClass ID: 0 Validation file: ensemble_iou_0.4._mAP_0.454_extracted_classes_0.csv\nRadiologist: R8 mAP: 0.991058\nRadiologist: R9 mAP: 0.954593\nRadiologist: R10 mAP: 0.971534\nRadiologist: R11 mAP: 0.379283\nRadiologist: R12 mAP: 0.019819\nRadiologist: R13 mAP: 0.511284\nRadiologist: R14 mAP: 0.527909\nRadiologist: R15 mAP: 0.279870\nRadiologist: R16 mAP: 0.506167\nRadiologist: R17 mAP: 0.447092\nInteresting, we have perfect match for R8, R9 and R10, but very poor on other radiologists. Bad thing is that there are too small entries by other radiologists in train:
\nR11 - Class: 0 Entries: 30 AP: 0.379283\nR12 - Class: 0 Entries: 12 AP: 0.019819\nR13 - Class: 0 Entries: 32 AP: 0.511284\nR14 - Class: 0 Entries: 74 AP: 0.527909\nR15 - Class: 0 Entries: 25 AP: 0.279870\nR16 - Class: 0 Entries: 23 AP: 0.506167\nR17 - Class: 0 Entries: 10 AP: 0.447092\nOne more thing that R8, R9 and R10 never markup with any other radiologists. And what if test set marked up by some other radiologists (may be even by R13, R14 etc) and making boxes too similar to these 3 we overfit to train? And what if with better models we went further from test mark up?
\nI checked by eyes images of class 0 in train marked up by R8, R9 and R10 and by rare radiologists to check the difference. I was lucky to see that other radiologists often use much larger boxes comparing to R8, R9 and R10. So I made a small trick by hand:
\nx1 -= 100 y2 += 100 for all boxes with class 0
\nAnd checked local validation, which improved a lot for other radiologists, while R8, R9 and R10 became worse:
Radiologist: R10 Class: 0 Entries: 2349 AP: 0.559458\nRadiologist: R11 Class: 0 Entries: 30 AP: 0.559355\nRadiologist: R12 Class: 0 Entries: 12 AP: 0.351739\nRadiologist: R13 Class: 0 Entries: 32 AP: 0.483730\nRadiologist: R14 Class: 0 Entries: 74 AP: 0.623019\nRadiologist: R15 Class: 0 Entries: 25 AP: 0.341545\nRadiologist: R16 Class: 0 Entries: 23 AP: 0.506167\nRadiologist: R17 Class: 0 Entries: 10 AP: 0.452887\nI made the same trick on our 0.301 submission and got 0.309 on LB. This showed us the right direction. We finetuned our models using only rare radiologists. And it improved LB score for all of models:
\nRetinaNet (ResNet101): 0.246 -> 0.264
\nRetinaNet (ResNet152): 0.222 -> 0.237
\nYolo_v5: 0.251 -> 0.271
\nEnsemble for all models including original moved us to Public LB > 0.330
At this point it became hard to add new models to ensemble without proper validation. So we decided to switch to single class improvements. It was much easier to find proper stop epochs for this case. Also training procedure became more complicated. We added NIH dataset boxes for classes where it was possible. We prepared NIH dataset in format of VinBigData Chest X-ray competition. We added it as separate radiologist R99. We also prepeared dataset for Pneumothorax, based on SIIM competition, but it didn't help.
\nWe used only one RetinaNet-R101 model for finetuning. We started from best previous weights obtained for multiclass case. So finetuning was pretty fast. We changed sampling in following way:
\nIn parallel we fine-tuned some models on very high resolution (2048px). Some classes from these models catch small boxes better than previous models. It improved our local validation.
\nThis approaches together allowed us to improve even more: >0.36 on public
Actually we decided to create 2 submissions: best on validation and best on LB. We made first submission pretty early. So at the late stage of competition we mostly work on LB-overfitting. That was a little bit risky, but after private LB was revealed we found out it was right thing to do. )
\nYou can find code for out solution on github:
\nhttps://github.com/ZFTurbo/2nd-place-solution-for-VinBigData-Chest-X-ray-Abnormalities-Detection
A small addition.
\nAt first, I trained using pictures with boxes only, and used postprocessing with a binary classifier using threshold to remove boxes. But then, following @zfturbo's example, I trained with empty images and just adding 14 class with the probability of being healthy.
\nIt is tricky to write a special dataset loader in Yolo v5, so I decided to triple every image with boxes. To accelerate training, I started with weights from previous training on WBF markup (the first stage). The second stage was training on rare radiologists only. I have also tried the following techniques:
\n• Choosing checkpoints for each class by the local validation. I see now it was profitable on private LB, but not so obvious on public LB.
\n• Using pseudo-labels from the test set. I used ~700 sure unhealthy images. I don’t know the impact of that alone since it was along with tripling together.
\n• Adding NIH dataset.
\n• Training with higher resolution (didn’t give an evident boost)
\nThe best Yolo model for public LB is 0.292, but it is not so good on private LB. The best Yolo for private LB is 0.288.
Congrats and thanks a lot for this detailed elaboration. Do you know the boost you got by blending in public solutions? It seems to help quite a bit as also shown by others.
\nWe did very similar investigation on class0 and came to the same conclusion as you.
\nWe mostly based our whole validation on rare annotators as those are way more prominent in test.
", "rawMarkdown": "Congrats and thanks a lot for this detailed elaboration. Do you know the boost you got by blending in public solutions? It seems to help quite a bit as also shown by others.\n\nWe did very similar investigation on class0 and came to the same conclusion as you.\n\nWe mostly based our whole validation on rare annotators as those are way more prominent in test.", "votes": 6, "replies": [ { "id": 1258040, "postDate": "2021-03-31T09:42:12.987Z", "content": "The score boost from public solutions is due to class 12 (it's the rarest class actually). It is ~0.03 on public LB, but not so good on private LB
", "rawMarkdown": "The score boost from public solutions is due to class 12 (it's the rarest class actually). It is ~0.03 on public LB, but not so good on private LB", "votes": 7 }, { "id": 1258043, "postDate": "2021-03-31T09:43:21.677Z", "content": "Thanks! As I remember public submissions catch rare 12 class [Pneumothorax] very good. I think there are only few (2-5?) boxes of Pneumothorax class on public. Looks like it can be the main source of shake up. )
\nYou can see it in my table (it was even better than our validation):
\nClass 12: 0.037 (mAP LB: 0.555 Valid: 0.388)
I see, we lacked subs to check all classes. What's the private score of that class 12 submission now?
", "rawMarkdown": "I see, we lacked subs to check all classes. What's the private score of that class 12 submission now?", "votes": 1 }, { "id": 1258054, "postDate": "2021-03-31T09:52:12.810Z", "content": "Class 12: 0.037 -> 0.013 (Note: as I remember it contains not only public solution, but ensembled with our models as well)
", "rawMarkdown": "Class 12: 0.037 -> 0.013 (Note: as I remember it contains not only public solution, but ensembled with our models as well)", "votes": 1 }, { "id": 1258071, "postDate": "2021-03-31T10:11:20.677Z", "content": "So most likely the public score difference came mostly from class 12 and the gap got smaller after class 12 did worse on private. Thanks!
", "rawMarkdown": "So most likely the public score difference came mostly from class 12 and the gap got smaller after class 12 did worse on private. Thanks!", "votes": 3 } ] }, { "id": 1258081, "postDate": "2021-03-31T10:20:43.280Z", "content": "Congrats for the second place and thanks for the solution
\nI have some questions @zfturbo :
\n\nHow do you use wbf on train.csv, do you consider each Radiologist as a \"model\" ?
\n
I just apply WBF to all boxes for given image with same confidence = 1. If boxes intersect with large IoU (>0.4) they fused in single box, otherwise stays \"as is\".
\n\n\nWhat is \"Std + Mirror images\", is this an data augmentation technique ?
\n
It's standard Test Time Augmentation. I used original image and mirrored image - then ensemble obtained boxes for these 2 images using WBF.
\n\n\nHow do you \"finetuned our models using only rare radiologists\" ?
\n
I removed images which were marked by R8, R9 and R10 radiologists from training and from validation. Finetuning started from best weights of these models trained on full data.
\n\n\nDo you train \"no fingding\" images with your OD models ? Since i note that you have class 14 mAP on Local Valid?
\n
From the begining for class 14 I used predictions from binary classifier (EffNetB5). But I don't remember what exactly we used in final submission. I think we changed it to some other method involving object detection models, because it was better on LB. Need to ask Sergey.
", "rawMarkdown": "> How do you use wbf on train.csv, do you consider each Radiologist as a \"model\" ?\n\nI just apply WBF to all boxes for given image with same confidence = 1. If boxes intersect with large IoU (>0.4) they fused in single box, otherwise stays \"as is\".\n\n> What is \"Std + Mirror images\", is this an data augmentation technique ?\n\nIt's standard Test Time Augmentation. I used original image and mirrored image - then ensemble obtained boxes for these 2 images using WBF. \n\n> How do you \"finetuned our models using only rare radiologists\" ? \n\nI removed images which were marked by R8, R9 and R10 radiologists from training and from validation. Finetuning started from best weights of these models trained on full data.\n\n> Do you train \"no fingding\" images with your OD models ? Since i note that you have class 14 mAP on Local Valid?\n\nFrom the begining for class 14 I used predictions from binary classifier (EffNetB5). But I don't remember what exactly we used in final submission. I think we changed it to some other method involving object detection models, because it was better on LB. Need to ask Sergey.", "votes": 3 }, { "id": 1258120, "postDate": "2021-03-31T11:15:16.053Z", "content": "Once (accidentally) we made WBF for class 14, there was the binary classifier and the result of an OD model. And we saw that this is better (on public LB) than a pure classifier. Then we used this variant. We need to check if there is an effect on private LB from this.
", "rawMarkdown": "Once (accidentally) we made WBF for class 14, there was the binary classifier and the result of an OD model. And we saw that this is better (on public LB) than a pure classifier. Then we used this variant. We need to check if there is an effect on private LB from this.", "votes": 3 }, { "id": 1258195, "postDate": "2021-03-31T12:38:21.280Z", "content": "Interesting. What was your best 14 class AP score on the private LB?
", "rawMarkdown": "Interesting. What was your best 14 class AP score on the private LB?" }, { "id": 1258254, "postDate": "2021-03-31T13:39:36.197Z", "content": "I've checked all our variants on private LB. And we used the right one :)
\n0.064 both private and public LB
\nIt is the variant with object detection injected. Although the boost from OD is quite small.
Thank you @zfturbo @sergeyzlobin
", "rawMarkdown": "Thank you @zfturbo @sergeyzlobin ", "votes": 1 }, { "id": 1258887, "postDate": "2021-04-01T01:31:15.047Z", "content": "When you use the OD model to detect the 14th class, do you set the bbox to 0 0 1 1 or something else?
", "rawMarkdown": "When you use the OD model to detect the 14th class, do you set the bbox to 0 0 1 1 or something else?" } ] }, { "id": 1258117, "postDate": "2021-03-31T11:13:31.777Z", "content": "Congratz on #2 Win. You were a worthy competitor for us.
", "rawMarkdown": "Congratz on #2 Win. You were a worthy competitor for us.", "votes": 3 }, { "id": 1258809, "postDate": "2021-03-31T22:47:22.013Z", "content": "Congratulations @zfturbo and team! Great detective work analyzing the classes. And great discovery that IOU < 0.4 worked well. We didn't try low IOU threshold values.
\nThanks for sharing your Ensemble Boxes GitHub repo, we used it and it is awesome!
", "rawMarkdown": "Congratulations @zfturbo and team! Great detective work analyzing the classes. And great discovery that IOU < 0.4 worked well. We didn't try low IOU threshold values.\n\nThanks for sharing your [Ensemble Boxes GitHub repo][1], we used it and it is awesome!\n\n[1]: https://github.com/ZFTurbo/Weighted-Boxes-Fusion", "votes": 1 }, { "id": 1258129, "postDate": "2021-03-31T11:23:05.523Z", "content": "@zfturbo amazing. Congrats. I had spent time on CenterNet. Wish I had seen that one all the way through!
", "rawMarkdown": "@zfturbo amazing. Congrats. I had spent time on CenterNet. Wish I had seen that one all the way through!", "votes": 1 }, { "id": 1258070, "postDate": "2021-03-31T10:11:19.870Z", "content": "Congratulations on the second place and thanks for the detailed explanation!
\nI got a lot from @zfturbo team's great work.
Congratulations and thanks a lot for the detailed solution! The investigation and your approach is so amazing!
", "rawMarkdown": "Congratulations and thanks a lot for the detailed solution! The investigation and your approach is so amazing!", "votes": 1 }, { "id": 1258051, "postDate": "2021-03-31T09:48:37.587Z", "content": "Congrats ! Your investigation on individual classes and radiologists is very strong
", "rawMarkdown": "Congrats ! Your investigation on individual classes and radiologists is very strong", "votes": 1 }, { "id": 1258055, "postDate": "2021-03-31T09:52:32.623Z", "content": "Congrats for the second place and thanks for the detailed writeup!
\nWe also noticed the systematic differences between annotators (https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229642)
\nI wonder if applying the bigger boxes finetune trick only for images not annotated by R8-R10 would further boost the score.
Congratulations for 2nd place and thanks for sharing detailed solution @zfturbo and team
", "rawMarkdown": "Congratulations for 2nd place and thanks for sharing detailed solution @zfturbo and team", "votes": 2 }, { "id": 1276625, "postDate": "2021-04-17T18:46:44.387Z", "content": "Congrats! Great work!
", "rawMarkdown": "Congrats! Great work!" }, { "id": 1260921, "postDate": "2021-04-02T14:34:18.100Z", "content": "Congratz, which repo did you use for *RetinaNet ? @zfturbo
", "rawMarkdown": "Congratz, which repo did you use for **RetinaNet* ? @zfturbo ", "replies": [ { "id": 1260965, "postDate": "2021-04-02T15:03:34.087Z", "content": "I still use this code:
\nhttps://github.com/ZFTurbo/Keras-RetinaNet-for-Open-Images-Challenge-2018
It's outdated a little bit )
", "rawMarkdown": "I still use this code:\nhttps://github.com/ZFTurbo/Keras-RetinaNet-for-Open-Images-Challenge-2018\n\nIt's outdated a little bit )", "votes": 1 } ] }, { "id": 1259469, "postDate": "2021-04-01T12:34:12.717Z", "content": "Congrats, @zfturbo ! Amazing work
", "rawMarkdown": "Congrats, @zfturbo ! Amazing work" }, { "id": 1259370, "postDate": "2021-04-01T10:43:26.160Z", "content": "\"The first thing we tried is to calculate validation mAP based on Radiologist ID.\"
\n=> How did you calculate the mAP for each Radiologist ? Did you just remove all the other radiologists' bbox in ground truth and calculate just the remained one ?
Yes, just use images with bboxes marked by given radiologist discarding all other radiologist for this image. I don't remember exactly, but probably I also used all empty images.
", "rawMarkdown": "Yes, just use images with bboxes marked by given radiologist discarding all other radiologist for this image. I don't remember exactly, but probably I also used all empty images.", "votes": 1 }, { "id": 1260258, "postDate": "2021-04-02T01:12:44.233Z", "content": "Thank you, and congrats on the Prize !
", "rawMarkdown": "Thank you, and congrats on the Prize !" }, { "id": 1260535, "postDate": "2021-04-02T07:52:31.503Z", "content": "\n\nI don't remember exactly, but probably I also used all empty images.
\n
Yes, empty images were included. That's why the validation was not fast. :(
", "rawMarkdown": "> I don't remember exactly, but probably I also used all empty images.\n\nYes, empty images were included. That's why the validation was not fast. :(", "votes": 1 } ] }, { "id": 1258035, "postDate": "2021-03-31T09:38:59.577Z", "content": "@zfturbo Congratulations on 2 nd Prize and Thanks for sharing the approach
", "rawMarkdown": "@zfturbo Congratulations on 2 nd Prize and Thanks for sharing the approach" } ], "comments": [ { "id": 1258660, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2021-03-31T19:37:33.237000", "content": "A small addition.
\nAt first, I trained using pictures with boxes only, and used postprocessing with a binary classifier using threshold to remove boxes. But then, following @zfturbo's example, I trained with empty images and just adding 14 class with the probability of being healthy.
\nIt is tricky to write a special dataset loader in Yolo v5, so I decided to triple every image with boxes. To accelerate training, I started with weights from previous training on WBF markup (the first stage). The second stage was training on rare radiologists only. I have also tried the following techniques:
\n• Choosing checkpoints for each class by the local validation. I see now it was profitable on private LB, but not so obvious on public LB.
\n• Using pseudo-labels from the test set. I used ~700 sure unhealthy images. I don’t know the impact of that alone since it was along with tripling together.
\n• Adding NIH dataset.
\n• Training with higher resolution (didn’t give an evident boost)
\nThe best Yolo model for public LB is 0.292, but it is not so good on private LB. The best Yolo for private LB is 0.288.
Congrats and thanks a lot for this detailed elaboration. Do you know the boost you got by blending in public solutions? It seems to help quite a bit as also shown by others.
\nWe did very similar investigation on class0 and came to the same conclusion as you.
\nWe mostly based our whole validation on rare annotators as those are way more prominent in test.
", "votes": 6, "replies": [ { "id": 1258040, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2021-03-31T09:42:12.987000", "content": "The score boost from public solutions is due to class 12 (it's the rarest class actually). It is ~0.03 on public LB, but not so good on private LB
", "votes": 7, "replies": [] }, { "id": 1258043, "author_name": "ZFTurbo", "author_url": "", "post_date": "2021-03-31T09:43:21.677000", "content": "Thanks! As I remember public submissions catch rare 12 class [Pneumothorax] very good. I think there are only few (2-5?) boxes of Pneumothorax class on public. Looks like it can be the main source of shake up. )
\nYou can see it in my table (it was even better than our validation):
\nClass 12: 0.037 (mAP LB: 0.555 Valid: 0.388)
I see, we lacked subs to check all classes. What's the private score of that class 12 submission now?
", "votes": 1, "replies": [] }, { "id": 1258054, "author_name": "ZFTurbo", "author_url": "", "post_date": "2021-03-31T09:52:12.810000", "content": "Class 12: 0.037 -> 0.013 (Note: as I remember it contains not only public solution, but ensembled with our models as well)
", "votes": 1, "replies": [] }, { "id": 1258071, "author_name": "Psi", "author_url": "", "post_date": "2021-03-31T10:11:20.677000", "content": "So most likely the public score difference came mostly from class 12 and the gap got smaller after class 12 did worse on private. Thanks!
", "votes": 3, "replies": [] } ] }, { "id": 1258081, "author_name": "Bùi Nhật Trường", "author_url": "", "post_date": "2021-03-31T10:20:43.280000", "content": "Congrats for the second place and thanks for the solution
\nI have some questions @zfturbo :
\n\nHow do you use wbf on train.csv, do you consider each Radiologist as a \"model\" ?
\n
I just apply WBF to all boxes for given image with same confidence = 1. If boxes intersect with large IoU (>0.4) they fused in single box, otherwise stays \"as is\".
\n\n\nWhat is \"Std + Mirror images\", is this an data augmentation technique ?
\n
It's standard Test Time Augmentation. I used original image and mirrored image - then ensemble obtained boxes for these 2 images using WBF.
\n\n\nHow do you \"finetuned our models using only rare radiologists\" ?
\n
I removed images which were marked by R8, R9 and R10 radiologists from training and from validation. Finetuning started from best weights of these models trained on full data.
\n\n\nDo you train \"no fingding\" images with your OD models ? Since i note that you have class 14 mAP on Local Valid?
\n
From the begining for class 14 I used predictions from binary classifier (EffNetB5). But I don't remember what exactly we used in final submission. I think we changed it to some other method involving object detection models, because it was better on LB. Need to ask Sergey.
", "votes": 3, "replies": [] }, { "id": 1258120, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2021-03-31T11:15:16.053000", "content": "Once (accidentally) we made WBF for class 14, there was the binary classifier and the result of an OD model. And we saw that this is better (on public LB) than a pure classifier. Then we used this variant. We need to check if there is an effect on private LB from this.
", "votes": 3, "replies": [] }, { "id": 1258195, "author_name": "beluga", "author_url": "", "post_date": "2021-03-31T12:38:21.280000", "content": "Interesting. What was your best 14 class AP score on the private LB?
", "votes": 0, "replies": [] }, { "id": 1258254, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2021-03-31T13:39:36.197000", "content": "I've checked all our variants on private LB. And we used the right one :)
\n0.064 both private and public LB
\nIt is the variant with object detection injected. Although the boost from OD is quite small.
Thank you @zfturbo @sergeyzlobin
", "votes": 1, "replies": [] }, { "id": 1258887, "author_name": "tik_boa", "author_url": "", "post_date": "2021-04-01T01:31:15.047000", "content": "When you use the OD model to detect the 14th class, do you set the bbox to 0 0 1 1 or something else?
", "votes": 0, "replies": [] } ] }, { "id": 1258117, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-03-31T11:13:31.777000", "content": "Congratz on #2 Win. You were a worthy competitor for us.
", "votes": 3, "replies": [] }, { "id": 1258809, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T22:47:22.013000", "content": "Congratulations @zfturbo and team! Great detective work analyzing the classes. And great discovery that IOU < 0.4 worked well. We didn't try low IOU threshold values.
\nThanks for sharing your Ensemble Boxes GitHub repo, we used it and it is awesome!
", "votes": 1, "replies": [] }, { "id": 1258129, "author_name": "Charlie Craine", "author_url": "", "post_date": "2021-03-31T11:23:05.523000", "content": "@zfturbo amazing. Congrats. I had spent time on CenterNet. Wish I had seen that one all the way through!
", "votes": 1, "replies": [] }, { "id": 1258070, "author_name": "Wang Xing", "author_url": "", "post_date": "2021-03-31T10:11:19.870000", "content": "Congratulations on the second place and thanks for the detailed explanation!
\nI got a lot from @zfturbo team's great work.
Congratulations and thanks a lot for the detailed solution! The investigation and your approach is so amazing!
", "votes": 1, "replies": [] }, { "id": 1258051, "author_name": "Matthieu Planté", "author_url": "", "post_date": "2021-03-31T09:48:37.587000", "content": "Congrats ! Your investigation on individual classes and radiologists is very strong
", "votes": 1, "replies": [] }, { "id": 1258055, "author_name": "beluga", "author_url": "", "post_date": "2021-03-31T09:52:32.623000", "content": "Congrats for the second place and thanks for the detailed writeup!
\nWe also noticed the systematic differences between annotators (https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229642)
\nI wonder if applying the bigger boxes finetune trick only for images not annotated by R8-R10 would further boost the score.
Congratulations for 2nd place and thanks for sharing detailed solution @zfturbo and team
", "votes": 2, "replies": [] }, { "id": 1276625, "author_name": "Sai Prakash", "author_url": "", "post_date": "2021-04-17T18:46:44.387000", "content": "Congrats! Great work!
", "votes": 0, "replies": [] }, { "id": 1260921, "author_name": "Awsaf", "author_url": "", "post_date": "2021-04-02T14:34:18.100000", "content": "Congratz, which repo did you use for *RetinaNet ? @zfturbo
", "votes": 0, "replies": [ { "id": 1260965, "author_name": "ZFTurbo", "author_url": "", "post_date": "2021-04-02T15:03:34.087000", "content": "I still use this code:
\nhttps://github.com/ZFTurbo/Keras-RetinaNet-for-Open-Images-Challenge-2018
It's outdated a little bit )
", "votes": 1, "replies": [] } ] }, { "id": 1259469, "author_name": "Sidney Ng", "author_url": "", "post_date": "2021-04-01T12:34:12.717000", "content": "Congrats, @zfturbo ! Amazing work
", "votes": 0, "replies": [] }, { "id": 1259370, "author_name": "Tung Vu", "author_url": "", "post_date": "2021-04-01T10:43:26.160000", "content": "\"The first thing we tried is to calculate validation mAP based on Radiologist ID.\"
\n=> How did you calculate the mAP for each Radiologist ? Did you just remove all the other radiologists' bbox in ground truth and calculate just the remained one ?
Yes, just use images with bboxes marked by given radiologist discarding all other radiologist for this image. I don't remember exactly, but probably I also used all empty images.
", "votes": 1, "replies": [] }, { "id": 1260258, "author_name": "Tung Vu", "author_url": "", "post_date": "2021-04-02T01:12:44.233000", "content": "Thank you, and congrats on the Prize !
", "votes": 0, "replies": [] }, { "id": 1260535, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2021-04-02T07:52:31.503000", "content": "\n\nI don't remember exactly, but probably I also used all empty images.
\n
Yes, empty images were included. That's why the validation was not fast. :(
", "votes": 1, "replies": [] } ] }, { "id": 1258035, "author_name": "Tensor Girl", "author_url": "", "post_date": "2021-03-31T09:38:59.577000", "content": "@zfturbo Congratulations on 2 nd Prize and Thanks for sharing the approach
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1258009": "## Part 1: Initial models\n\nAt first I started to try different OD models for this dataset. I tried: EffDet (B5), CenterNet (HourGlass), Yolo_v5, Retinanet (RsNet50, ResNet101, ResNet152), Faster RCNN\n\nMain initial tricks:\n1. To train models I used [WBF](https://github.com/ZFTurbo/Weighted-Boxes-Fusion) on train.csv with IoU: 0.4. I usually used this file with most of models for training and for validation.\n2. Using validation I find out that it’s critical to use lower values for IoU in NMS which filters boxes on output of models. So I used NMS IoU: 0.25 for Yolo_v5 and NMS IoU: 0.3 for RetinaNet. Looks like, it was better for high confident boxes to consume more boxes around it.\n3. I didn’t want to change anchors for RetinaNet so I created version of train.csv with boxes which is close to square, because default Retina doesn’t like elongated boxes.\n\nBest models were the following:\n- Yolo_v5: ~0.251 Public LB (Std + Mirror images 5KFold, 640px)\n- RetinaNet (ResNet101): ~0.246 Public LB (Std + Mirror images 5KFold, 1024px)\n- RetinaNet (ResNet152): ~0.222 Public LB (Std + Mirror images 5KFold, 800px)\n- CenterNet (HourGlass): ~0.196 Public LB(Std + Mirror images 5KFold, 512px)\n\nEffDet – same code as Wheat competition, but works pretty poor on this dataset. With Faster-RCNN I think I have some problem with code, because it has the very bad score.\n\nUsing WBF ensemble of these models + 2 public models (Detectron2 and Yolo_v5):\n- 0.221: https://www.kaggle.com/corochann/vinbigdata-detectron2-prediction\n- 0.204: https://www.kaggle.com/awsaf49/vinbigdata-2-class-filter?scriptVersionId=51562634\n\nI was able to achieve 0.300 on Public LB (0.289 Private LB). Then I merge with my teammates. They had more mmdetection models as well as Yolo_v5, but together we were able to got only 0.301.\n\n## Part 2: Investigation\nWhile we were stuck we decided to get individual score for each class and compare with our OOF validation. For 3 days we got the following table:\n\n```\nClass 0: 0.014 (mAP LB: 0.210 Valid: 0.900)\nClass 1: 0.009 (mAP LB: 0.135 Valid: 0.335)\nClass 2: 0.017 (mAP LB: 0.255 Valid: 0.278)\nClass 3: 0.046 (mAP LB: 0.690 Valid: 0.915)\nClass 4: 0.019 (mAP LB: 0.285 Valid: 0.438)\nClass 5: 0.013 (mAP LB: 0.195 Valid: 0.376)\nClass 6: 0.010 (mAP LB: 0.150 Valid: 0.465)\nClass 7: 0.003 (mAP LB: 0.045 Valid: 0.400)\nClass 8: 0.011 (mAP LB: 0.165 Valid: 0.372)\nClass 9: 0.004 (mAP LB: 0.060 Valid: 0.162)\nClass 10: 0.027 (mAP LB: 0.405 Valid: 0.548)\nClass 11: 0.012 (mAP LB: 0.180 Valid: 0.343)\nClass 12: 0.037 (mAP LB: 0.555 Valid: 0.388)\nClass 13: 0.009 (mAP LB: 0.135 Valid: 0.432)\nClass 14: 0.063 (mAP LB: 0.945 Valid: 0.772)\nSum: 0.294\n```\nAs it can be seen LB mAP the very different from local mAP. We started from investigation of Class 0 difference (it was most suspicious). We have near perfect prediction on validation while LB score is so small. The first thing we tried is to calculate validation mAP based on Radiologist ID.\n```\nClass ID: 0 Validation file: ensemble_iou_0.4._mAP_0.454_extracted_classes_0.csv\nRadiologist: R8 mAP: 0.991058\nRadiologist: R9 mAP: 0.954593\nRadiologist: R10 mAP: 0.971534\nRadiologist: R11 mAP: 0.379283\nRadiologist: R12 mAP: 0.019819\nRadiologist: R13 mAP: 0.511284\nRadiologist: R14 mAP: 0.527909\nRadiologist: R15 mAP: 0.279870\nRadiologist: R16 mAP: 0.506167\nRadiologist: R17 mAP: 0.447092\n```\n\nInteresting, we have perfect match for R8, R9 and R10, but very poor on other radiologists. Bad thing is that there are too small entries by other radiologists in train:\n```\nR11 - Class: 0 Entries: 30 AP: 0.379283\nR12 - Class: 0 Entries: 12 AP: 0.019819\nR13 - Class: 0 Entries: 32 AP: 0.511284\nR14 - Class: 0 Entries: 74 AP: 0.527909\nR15 - Class: 0 Entries: 25 AP: 0.279870\nR16 - Class: 0 Entries: 23 AP: 0.506167\nR17 - Class: 0 Entries: 10 AP: 0.447092\n```\nOne more thing that R8, R9 and R10 never markup with any other radiologists. And what if test set marked up by some other radiologists (may be even by R13, R14 etc) and making boxes too similar to these 3 we overfit to train? And what if with better models we went further from test mark up?\n\nI checked by eyes images of class 0 in train marked up by R8, R9 and R10 and by rare radiologists to check the difference. I was lucky to see that other radiologists often use much larger boxes comparing to R8, R9 and R10. So I made a small trick by hand:\n`x1 -= 100 y2 += 100 for all boxes with class 0 `\nAnd checked local validation, which improved a lot for other radiologists, while R8, R9 and R10 became worse:\n```\nRadiologist: R10 Class: 0 Entries: 2349 AP: 0.559458\nRadiologist: R11 Class: 0 Entries: 30 AP: 0.559355\nRadiologist: R12 Class: 0 Entries: 12 AP: 0.351739\nRadiologist: R13 Class: 0 Entries: 32 AP: 0.483730\nRadiologist: R14 Class: 0 Entries: 74 AP: 0.623019\nRadiologist: R15 Class: 0 Entries: 25 AP: 0.341545\nRadiologist: R16 Class: 0 Entries: 23 AP: 0.506167\nRadiologist: R17 Class: 0 Entries: 10 AP: 0.452887\n```\nI made the same trick on our 0.301 submission and got 0.309 on LB. This showed us the right direction. We finetuned our models using only rare radiologists. And it improved LB score for all of models:\n\nRetinaNet (ResNet101): 0.246 -> 0.264\nRetinaNet (ResNet152): 0.222 -> 0.237\nYolo_v5: 0.251 -> 0.271\nEnsemble for all models including original moved us to Public LB > 0.330\n\n## Part 3 (Single classes):\n\nAt this point it became hard to add new models to ensemble without proper validation. So we decided to switch to single class improvements. It was much easier to find proper stop epochs for this case. Also training procedure became more complicated. We added NIH dataset boxes for classes where it was possible. We prepared [NIH dataset in format of VinBigData Chest X-ray competition](https://www.kaggle.com/zfturbo/nih-chest-xray-dataset-bbox-for-vinbigdata). We added it as separate radiologist R99. We also prepeared [dataset for Pneumothorax, based on SIIM competition](https://www.kaggle.com/zfturbo/siimacr-pneumothorax-for-vinbigdata-chest-xray), but it didn't help. \n\nWe used only one RetinaNet-R101 model for finetuning. We started from best previous weights obtained for multiclass case. So finetuning was pretty fast. We changed sampling in following way:\n- We first select random radiologist, then select random class (target_class ot empty image), then select random image which were marked by selected radiologist and selected class.\n- Validation was also slightly changed. We excluded all non-empty images which were marked by R8, R9 and R10 radiologists.\n\nIn parallel we fine-tuned some models on very high resolution (2048px). Some classes from these models catch small boxes better than previous models. It improved our local validation.\nThis approaches together allowed us to improve even more: >0.36 on public\n\n## Part 4. Final steps\n\nActually we decided to create 2 submissions: best on validation and best on LB. We made first submission pretty early. So at the late stage of competition we mostly work on LB-overfitting. That was a little bit risky, but after private LB was revealed we found out it was right thing to do. )\n\n### Models we choose.\n- Best at local validation (around 0.51 mAP locally), Public: 0.304, private: 0.304\n- Best at LB. Public: 0.365, private: 0.307\n- Our best model at private has 0.319 score, but we had no chance to choose it because it had too low score on public - only 0.292. And it wasn't best at local validation.\n\n## Code\n\nYou can find code for out solution on github:\nhttps://github.com/ZFTurbo/2nd-place-solution-for-VinBigData-Chest-X-ray-Abnormalities-Detection\n", "1258660": "A small addition.\n\nAt first, I trained using pictures with boxes only, and used postprocessing with a binary classifier using threshold to remove boxes. But then, following @zfturbo's example, I trained with empty images and just adding 14 class with the probability of being healthy.\n\nIt is tricky to write a special dataset loader in Yolo v5, so I decided to triple every image with boxes. To accelerate training, I started with weights from previous training on WBF markup (the first stage). The second stage was training on rare radiologists only. I have also tried the following techniques:\n•\tChoosing checkpoints for each class by the local validation. I see now it was profitable on private LB, but not so obvious on public LB.\n•\tUsing pseudo-labels from the test set. I used ~700 sure unhealthy images. I don’t know the impact of that alone since it was along with tripling together. \n•\tAdding NIH dataset.\n•\tTraining with higher resolution (didn’t give an evident boost)\nThe best Yolo model for public LB is 0.292, but it is not so good on private LB. The best Yolo for private LB is 0.288.\n\n", "1258027": "Congrats and thanks a lot for this detailed elaboration. Do you know the boost you got by blending in public solutions? It seems to help quite a bit as also shown by others.\n\nWe did very similar investigation on class0 and came to the same conclusion as you.\n\nWe mostly based our whole validation on rare annotators as those are way more prominent in test.", "1258081": "Congrats for the second place and thanks for the solution\nI have some questions @zfturbo :\n1. How do you use wbf on train.csv, do you consider each Radiologist as a \"model\" ?\n2. What is \"Std + Mirror images\", is this an data augmentation technique ?\n3. How do you \"finetuned our models using only rare radiologists\" ? \n4. Do you train \"no fingding\" images with your OD models ? Since i note that you have class 14 mAP on Local Valid ?", "1258117": "Congratz on #2 Win. You were a worthy competitor for us.", "1258809": "Congratulations @zfturbo and team! Great detective work analyzing the classes. And great discovery that IOU < 0.4 worked well. We didn't try low IOU threshold values.\n\nThanks for sharing your [Ensemble Boxes GitHub repo][1], we used it and it is awesome!\n\n[1]: https://github.com/ZFTurbo/Weighted-Boxes-Fusion", "1258129": "@zfturbo amazing. Congrats. I had spent time on CenterNet. Wish I had seen that one all the way through!", "1258070": "Congratulations on the second place and thanks for the detailed explanation!\nI got a lot from @zfturbo team's great work.", "1258058": "Congratulations and thanks a lot for the detailed solution! The investigation and your approach is so amazing!", "1258051": "Congrats ! Your investigation on individual classes and radiologists is very strong", "1258055": "Congrats for the second place and thanks for the detailed writeup! \n\n\nWe also noticed the systematic differences between annotators (https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229642)\nI wonder if applying the bigger boxes finetune trick only for images not annotated by R8-R10 would further boost the score.", "1258038": "Congratulations for 2nd place and thanks for sharing detailed solution @zfturbo and team", "1276625": "Congrats! Great work!", "1260921": "Congratz, which repo did you use for **RetinaNet* ? @zfturbo ", "1259469": "Congrats, @zfturbo ! Amazing work", "1259370": "\"The first thing we tried is to calculate validation mAP based on Radiologist ID.\"\n=> How did you calculate the mAP for each Radiologist ? Did you just remove all the other radiologists' bbox in ground truth and calculate just the remained one ? ", "1258035": "@zfturbo Congratulations on 2 nd Prize and Thanks for sharing the approach" } }