\n
![\"\"](\"https://i.imgur.com/BkzsbeP.png\")
Let's talk about a two-week adventure in the black-and-white world of lung.
\nWe had 2 classification models:
\np_box * 0.9 + p_cls * 0.1, where p_box is a probability of the box itself, and p_cls is the probability of the corresponding class for the whole imageBoth models were using ResNet34 as a backbone and standard augmentations: HorizontalFlip, RandomBrightnessContrast, ShiftScaleRotate. Binary classifier was achieving 0.992 ROC AUC locally, however got only 0.059 mAP submitted separately on the LB. So, probably, our classifiers were quite weak, that's why in the final models we've ensembled binary classifier with publicly available ones.
\nPreprocessing did not produce any results. We tried
\nThe universenet50-gfl universenet101-gfl models gave the greatest contribution. Probably Focal Loss is not bad for this task.
\nBest model: univernet50-gfl on all data (Different radiologist) with 35% No finding give LB 0.288 / 0.281 . CV 0.492 with all data and 0.54 without no finding on GT
Take NIH dataset and run ensemble.
\nBring in the appropriate classes:
\n'Hernia' -> 'Aortic enlargement'.
\nWe kept only the bounding box classes that matched the nih labels
\nLight class balance -- take 31000 images with pseudo labels from NiH
\nFinetune some folds on it give 0.001 to ensemble. It gives boost 9, 11,12 classes
Use some kind of voting. It depends on score, IOU score and unique bbox. Unique part just calculate how many models votes for current bbox (its better then pre nms all results to make sure that model not give multi bbox in position with current nms). Its boost No finding images, if we are not sure about the classifier:
\nvoting 0.502 all / 0.564 val only on gt
\nuniq voting 0.522 all / 0.56 val only on gt
The picture started to come together towards the end and so only a fraction of the cool stuff got done. Best of the unpublished private 0.317 (public 0.309)
\nBest too late moments:
Let's talk about a two-week adventure in the black-and-white world of lung.
\nWe had 2 classification models:
\np_box * 0.9 + p_cls * 0.1, where p_box is a probability of the box itself, and p_cls is the probability of the corresponding class for the whole imageBoth models were using ResNet34 as a backbone and standard augmentations: HorizontalFlip, RandomBrightnessContrast, ShiftScaleRotate. Binary classifier was achieving 0.992 ROC AUC locally, however got only 0.059 mAP submitted separately on the LB. So, probably, our classifiers were quite weak, that's why in the final models we've ensembled binary classifier with publicly available ones.
\nPreprocessing did not produce any results. We tried
\nThe universenet50-gfl universenet101-gfl models gave the greatest contribution. Probably Focal Loss is not bad for this task.
\nBest model: univernet50-gfl on all data (Different radiologist) with 35% No finding give LB 0.288 / 0.281 . CV 0.492 with all data and 0.54 without no finding on GT
Take NIH dataset and run ensemble.
\nBring in the appropriate classes:
\n'Hernia' -> 'Aortic enlargement'.
\nWe kept only the bounding box classes that matched the nih labels
\nLight class balance -- take 31000 images with pseudo labels from NiH
\nFinetune some folds on it give 0.001 to ensemble. It gives boost 9, 11,12 classes
Use some kind of voting. It depends on score, IOU score and unique bbox. Unique part just calculate how many models votes for current bbox (its better then pre nms all results to make sure that model not give multi bbox in position with current nms). Its boost No finding images, if we are not sure about the classifier:
\nvoting 0.502 all / 0.564 val only on gt
\nuniq voting 0.522 all / 0.56 val only on gt
The picture started to come together towards the end and so only a fraction of the cool stuff got done. Best of the unpublished private 0.317 (public 0.309)
\nBest too late moments:
Good job, congrats @nicksergievskiy @golubev and @ybabakhin on 8th place. Thanks for sharing detailed solution!
", "rawMarkdown": "Good job, congrats @nicksergievskiy @golubev and @ybabakhin on 8th place. Thanks for sharing detailed solution!", "votes": 2 } ], "comments": [ { "id": 1259433, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-04-01T12:10:09.777000", "content": "Good job, congrats @nicksergievskiy @golubev and @ybabakhin on 8th place. Thanks for sharing detailed solution!
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1259217": "Let's talk about a two-week adventure in the black-and-white world of lung.\n\n## Classification\nWe had 2 classification models:\n* Binary classification for 14 / non-14 class. It was used for the postprocessing / filtering of the object detection predictions\n* Multilabel classification for 14 disease classes. It was used to calculate the final box probability as: `p_box * 0.9 + p_cls * 0.1`, where `p_box` is a probability of the box itself, and `p_cls` is the probability of the corresponding class for the whole image\n\nBoth models were using ResNet34 as a backbone and standard augmentations: HorizontalFlip, RandomBrightnessContrast, ShiftScaleRotate. Binary classifier was achieving 0.992 ROC AUC locally, however got only 0.059 mAP submitted separately on the LB. So, probably, our classifiers were quite weak, that's why in the final models we've ensembled binary classifier with publicly available ones.\n\n## Preprocessing\nPreprocessing did not produce any results. We tried\n- Equalization\n- [Graham's-preprocessing]( https://www.kaggle.com/ratthachat/aptos-eye-preprocessing-in-diabetic-retinopathy#2.-Try-Ben-Graham's-preprocessing-method.)\n- combine preprocessing options into different channels\n\n\n\n\n\n\n## Preparing training data:\n- Remove duplicates\n- Different radiologist at different images\n- R11-17 (rare) radiologist\n- Add 35% No finding\n\n\n\n## Detector\n- [Universenet](https://github.com/shinya7y/UniverseNet)\n-- universenet50-gfl\n-- universenet101-gfl\n- FasterRCNN R101\n- YoloV5\n\n\nThe universenet50-gfl universenet101-gfl models gave the greatest contribution. Probably Focal Loss is not bad for this task.\nBest model: univernet50-gfl on all data (Different radiologist) with 35% No finding give **LB 0.288 / 0.281** . CV 0.492 with all data and 0.54 without no finding on GT\n\n## Extra data for pseudo labeling\nTake NIH dataset and run ensemble.\nBring in the appropriate classes:\n'Hernia' -> 'Aortic enlargement'. \nWe kept only the bounding box classes that matched the nih labels\nLight class balance -- take 31000 images with pseudo labels from NiH\nFinetune some folds on it give 0.001 to ensemble. It gives boost 9, 11,12 classes\n\n## BBox aggregation:\nUse some kind of [voting](https://github.com/dereyly/signate_tobaco/blob/b458e4fbd312b186ed0cac92321486b7d2fce175/data_utils/boxes_mask.py#L394). It depends on score, IOU score and unique bbox. Unique part just calculate how many models votes for current bbox (its better then pre nms all results to make sure that model not give multi bbox in position with current nms). Its boost No finding images, if we are not sure about the classifier:\nvoting 0.502 all / 0.564 val only on gt\nuniq voting 0.522 all / 0.56 val only on gt\n\n\n\nThe picture started to come together towards the end and so only a fraction of the cool stuff got done. Best of the unpublished private 0.317 (public 0.309)\nBest too late moments:\n- no filtering by class 14 -- +0.08\n- more models with rare R11-17 (less models with all data) in final ensemble -- +0.09\nThanks to the team @ybabakhin @golubev for this trip", "1259433": "Good job, congrats @nicksergievskiy @golubev and @ybabakhin on 8th place. Thanks for sharing detailed solution!" } }