![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2022/map1.png\")
I would like to thank my teammates @matthieuplante @seb6084 for inviting me onto their team. I have been wanting to learn more about object detection and my teammates were the perfect teachers.
\nWhen i joined them 1 week ago, they already had full pipelines to train VFNet, YoloV5, EfficientDet4 and more! Together we tuned the models, built some classifier models, and ensembled everything using weighted box fusion and Bayesian statistics!
\nOur final solution was an ensemble of three object detection models:
\nand three 15 class classifiers, i.e. 15 binary outs, with ensemble \"no finding\" CV AUC 0.9934:
\nAll decisions on how to ensemble folds and models. And how to combine object detection models with classifier models were determined via local validation.
\nWith ensemble, we achieved a validation score of 0.488 mAP, public LB 0.272, and private LB 0.296. To simulate the test set with our validation set, we randomly only kept one doctor's predictions per class per image otherwise validation would not be like LB scoring.
\nFor a given class, every bbox confidence score you predict needs to be ordered from most likely to least likely. Just like AUC metric. This is not per image, this is per CSV file for each class. If you predict a total of 30,000 bbox for class 0, then all 30,000 bbox confidence scores need to be correctly ordered from most likely to be class 0 to least likely to be class 0.
\nBy simply adjusting the confidence scores of your bbox and not changing their xmin, xmax, ymin, ymax you can increase your CV and LB score! In the following example, the entire plot is 1 class, each dot is 1 bbox and the numbers are confidence scores. If the confidence score for bbox D and G are changed then the mAP for this class increases by +0.12 !! Therefore it is important to calibrate bbox probabilities.
\n
![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2022/map2.png\")
If you add new bbox with confidence score less than the lowest confidence score for all of that class in your entire CSV file, then your CV LB can only increase, it cannot decrease! Therefore instead of removing bbox it is better to calibrate probabilities, i.e. lower probability for unlikely bbox.
\n![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2022/map3.png\")
The most effective way to train an object detection model is to remove all the images without any bbox. After doing this, your object detection model will predict a confidence score equal to
\n$$ \\text{confidence score} = \\text{prob}(class \\text{ given } finding = \\text{True})$$
\nTherefore to get the calibrated probability, i.e correct confidence score for classes 0 thru 13, you must use conditional probability rule and multiply by prob(finding) = 1 - prob(no finding). So we use our classifier and the formula
\n$$ \\text{confidence score} = \\text{prob}(class \\text{ given }finding = \\text{True}) * \\text{prob}(finding =\\text{True}) $$
\nThe first prob comes from object detection model and the second prob comes from classifier model ensemble. Using CV we discovered you can improve mAP metric with a 15 class classifier and the following formula:
\n$$ \\text{score} = base * \\text{prob}(class)^{0.1} * \\text{prob}(finding)^{0.1} $$
\nwhere base = prob(class given finding = True) that comes from your object detection model. And prob(class) is the probability that specific class is present on image from classifier model. And prob(finding) is the probability that image has a finding from classifier model.
There is nothing fancy about adding predictions for class 14. For each image just add the string 14 PR 0 0 1 1 where PR = prob(no finding) from your classifier. If your classifier is perfect, then you will score full mAP = 1.0 for class 14. Most classifiers have CV AUC 0.99+, so most teams will maximize this class mAP
I would like to thank my teammates @matthieuplante @seb6084 for inviting me onto their team. I have been wanting to learn more about object detection and my teammates were the perfect teachers.
\nWhen i joined them 1 week ago, they already had full pipelines to train VFNet, YoloV5, EfficientDet4 and more! Together we tuned the models, built some classifier models, and ensembled everything using weighted box fusion and Bayesian statistics!
\nOur final solution was an ensemble of three object detection models:
\nand three 15 class classifiers, i.e. 15 binary outs, with ensemble \"no finding\" CV AUC 0.9934:
\nAll decisions on how to ensemble folds and models. And how to combine object detection models with classifier models were determined via local validation.
\nWith ensemble, we achieved a validation score of 0.488 mAP, public LB 0.272, and private LB 0.296. To simulate the test set with our validation set, we randomly only kept one doctor's predictions per class per image otherwise validation would not be like LB scoring.
\nFor a given class, every bbox confidence score you predict needs to be ordered from most likely to least likely. Just like AUC metric. This is not per image, this is per CSV file for each class. If you predict a total of 30,000 bbox for class 0, then all 30,000 bbox confidence scores need to be correctly ordered from most likely to be class 0 to least likely to be class 0.
\nBy simply adjusting the confidence scores of your bbox and not changing their xmin, xmax, ymin, ymax you can increase your CV and LB score! In the following example, the entire plot is 1 class, each dot is 1 bbox and the numbers are confidence scores. If the confidence score for bbox D and G are changed then the mAP for this class increases by +0.12 !! Therefore it is important to calibrate bbox probabilities.
\n
If you add new bbox with confidence score less than the lowest confidence score for all of that class in your entire CSV file, then your CV LB can only increase, it cannot decrease! Therefore instead of removing bbox it is better to calibrate probabilities, i.e. lower probability for unlikely bbox.
\n
The most effective way to train an object detection model is to remove all the images without any bbox. After doing this, your object detection model will predict a confidence score equal to
\n$$ \\text{confidence score} = \\text{prob}(class \\text{ given } finding = \\text{True})$$
\nTherefore to get the calibrated probability, i.e correct confidence score for classes 0 thru 13, you must use conditional probability rule and multiply by prob(finding) = 1 - prob(no finding). So we use our classifier and the formula
\n$$ \\text{confidence score} = \\text{prob}(class \\text{ given }finding = \\text{True}) * \\text{prob}(finding =\\text{True}) $$
\nThe first prob comes from object detection model and the second prob comes from classifier model ensemble. Using CV we discovered you can improve mAP metric with a 15 class classifier and the following formula:
\n$$ \\text{score} = base * \\text{prob}(class)^{0.1} * \\text{prob}(finding)^{0.1} $$
\nwhere base = prob(class given finding = True) that comes from your object detection model. And prob(class) is the probability that specific class is present on image from classifier model. And prob(finding) is the probability that image has a finding from classifier model.
There is nothing fancy about adding predictions for class 14. For each image just add the string 14 PR 0 0 1 1 where PR = prob(no finding) from your classifier. If your classifier is perfect, then you will score full mAP = 1.0 for class 14. Most classifiers have CV AUC 0.99+, so most teams will maximize this class mAP
Brilliant! From the melonoma classification, I have learned a lot from your discussion. thank you
\nI have a question why your team trained the models with 1024 but inferred with lower resolutions?
", "rawMarkdown": "Brilliant! From the melonoma classification, I have learned a lot from your discussion. thank you\n\nI have a question why your team trained the models with 1024 but inferred with lower resolutions?", "votes": 5 }, { "id": 1263240, "postDate": "2021-04-05T08:30:04.340Z", "content": "Hi @cdeotte , thank for sharing your post-processing approach. I have a question. You said the accuracy will increase if the boxes confidence score increase. In the Figure 2, how about if all confidence score all increases 0.1, so all the position from A-G still remains same as Figure 1 right, so is it make any change different from Figure 1?
", "rawMarkdown": "Hi @cdeotte , thank for sharing your post-processing approach. I have a question. You said the accuracy will increase if the boxes confidence score increase. In the Figure 2, how about if all confidence score all increases 0.1, so all the position from A-G still remains same as Figure 1 right, so is it make any change different from Figure 1?\n", "votes": 1, "replies": [ { "id": 1263638, "postDate": "2021-04-05T14:57:29.507Z", "content": "No, then mAP will not increase. We increase mAP by rearranging the ranking order into a more accurate order just like the metric AUC.
Awesome post-processing approachโฆ as always :D
\nJust one question: do you think this approach is more robust and just generally works better than simply training with some empty images, and not using classifiers to calibrate confidence scores for each box?
", "rawMarkdown": "Awesome post-processing approach... as always :D \n\nJust one question: do you think this approach is more robust and just generally works better than simply training with some empty images, and not using classifiers to calibrate confidence scores for each box? ", "votes": 1, "replies": [ { "id": 1259671, "postDate": "2021-04-01T15:15:21.803Z", "content": "Great question @ivanpan . Note that this post process is not the same as training object detection with all images.
\nBayesian statistics say we should use score = base * prob(finding) where base is the probability from object detection training on only \"finding\" images. I believe this formula would produce the same bbox confidence scores as training an object detection on all images.
The above formula is equivilent (i.e. same rank order, same mAP) as using score = base**0.5 * prob(finding)**0.5. This is the geometric mean between the two. Now, base is over zealous because it was trained on only positive images, and multiplying by prob(finding) makes the result realistic.
Our final formula is equivilent to score = base**0.84 * prob(finding)**0.16. This is the geometric average where we give more weight to over zealous bbox confidence scores which optimizes the loop holes of the mAP metric.
To clarify further, our formula is score = base * prob(finding)**0.1 * prob(class)**0.1 which is more or less score = base * prob(finding)**0.2 which is equivalent (same ranking ordering, same mAP) to score = base**0.84 * prob(finding)**0.16 if you raise the right side to the exponent 5/6.
Hm, interesting. I actually understand the math perfectly (+ saw your response about Xu's solution), but it doesn't seem intuitive at all :)
", "rawMarkdown": "Hm, interesting. I actually understand the math perfectly (+ saw your response about Xu's solution), but it doesn't seem intuitive at all :) ", "votes": 1 }, { "id": 1259801, "postDate": "2021-04-01T17:08:19.173Z", "content": "Now that I think about it, it's actually more or less okay. Since your ending formula is pretty much the same as base**0.84 * prob(finding)**0.16, which is just geometric mean. But how to get 0.84/0.16? Grid search on [0, 1] with step 0.01 + check validation scores?
Yes, we found the formula using grid search on local validation.
\nYour question has me wondering how similar or different our post process is to training object detection on all images. As i said above, i think they are different and i think if i had an object detection model trained on all images, i could post process the confidence scores with a new formula not published anywhere that could increase it's mAP CV LB.
\nBecause the mAP metric rewards both precision and recall. And the best way to increase recall is to make more bbox on the images that we believe have findings (i.e. prob(finding)>0.5) and mAP does not penalize you for trying to find more bbox and being wrong.
Okay, it's official: my goal for 2021-2022 is to get into a team with you. I take models, you take post-processing ๐ช
", "rawMarkdown": "Okay, it's official: my goal for 2021-2022 is to get into a team with you. I take models, you take post-processing ๐ช" } ] }, { "id": 1258818, "postDate": "2021-03-31T23:08:24.803Z", "content": "Very well explained @cdeotte ! Congrats once again !
", "rawMarkdown": "Very well explained @cdeotte ! Congrats once again !", "votes": 1 }, { "id": 1258729, "postDate": "2021-03-31T20:55:16.780Z", "content": "Congrats! thanks for sharing insights about competition metric in detailed explanation.
\nI could not understand how to calibrate probability in \"Order2\" way, isn't it possible to calibrate only D & G class without knowing which is the true class??
", "rawMarkdown": "Congrats! thanks for sharing insights about competition metric in detailed explanation.\n\nI could not understand how to calibrate probability in \"Order2\" way, isn't it possible to calibrate only D & G class without knowing which is the true class??", "votes": 1, "replies": [ { "id": 1258795, "postDate": "2021-03-31T22:23:05.167Z", "content": "Thanks. The letters are not classes. The entire picture is a single class. Each letter is one bbox. The numbers are the confidence scores assigned to a bbox. And the bboxes are all bboxes in your entire submission.csv file for the 1 pictured class. (This is a made up example with few bbox).
So given 1 class with its many bboxes in our submitted CSV file, if we can modify all the confidence scores in a smart way, then we can increase that 1 class's mAP.
\n\nif we can modify all the confidence scores in a smart way
\n
Thanks for reply, but I'm sorry that I still could not understand how to achieve this. Order does not affect to the metric, so just multiplying some value to all confidence does not change the metric.
\nThen, how you can change the order of confidence scores without knowing ground truth \"truth=0 or 1\"?
Congrats! I admire your post-processing very much.๐
", "rawMarkdown": "Congrats! I admire your post-processing very much.๐", "votes": 1 }, { "id": 1258221, "postDate": "2021-03-31T13:06:52.853Z", "content": "Congrats and thanks for sharing your details! I have learned a lot from you. Thanks you.
", "rawMarkdown": "Congrats and thanks for sharing your details! I have learned a lot from you. Thanks you.", "votes": 1 }, { "id": 1257893, "postDate": "2021-03-31T07:30:21.377Z", "content": "Congrats!! And thanks for the sharing in details, you always did a very fascinating job in post-process
", "rawMarkdown": "Congrats!! And thanks for the sharing in details, you always did a very fascinating job in post-process ", "votes": 1, "replies": [ { "id": 1258367, "postDate": "2021-03-31T14:54:37.763Z", "content": "Thanks Tuan
", "rawMarkdown": "Thanks Tuan", "votes": 1 } ] }, { "id": 1257822, "postDate": "2021-03-31T05:59:54.567Z", "content": "Congratulations on the standing! It is people like you and your team whom really inspire beginner like myself to improve and become better. To have that intuition and experience to apply theories i never thought i could apply out of school ( Bayesian statistics ) ๐
", "rawMarkdown": "Congratulations on the standing! It is people like you and your team whom really inspire beginner like myself to improve and become better. To have that intuition and experience to apply theories i never thought i could apply out of school ( Bayesian statistics ) ๐", "votes": 1, "replies": [ { "id": 1257850, "postDate": "2021-03-31T06:31:29.913Z", "content": "Thanks. Congratulations on earning a bronze medal, top 10%
", "rawMarkdown": "Thanks. Congratulations on earning a bronze medal, top 10%", "votes": 1 } ] }, { "id": 1257801, "postDate": "2021-03-31T05:35:54.403Z", "content": "Congrats for the good models and stable validation results! I like the postprocessing ideas!
\nI wonder if you tested your Class 14 predictions separately by just submitting the 1 pixel boxes what would be the LB AP score?
", "rawMarkdown": "Congrats for the good models and stable validation results! I like the postprocessing ideas!\n\nI wonder if you tested your Class 14 predictions separately by just submitting the 1 pixel boxes what would be the LB AP score?", "votes": 1, "replies": [ { "id": 1257817, "postDate": "2021-03-31T05:53:28.913Z", "content": "Thanks. We did not during the competition, but i just submitted now. Our ensembled classifier output for finding / no finding achieves CV AUC 0.9934. And when submitted by itself achieves LB 0.064
", "rawMarkdown": "Thanks. We did not during the competition, but i just submitted now. Our ensembled classifier output for finding / no finding achieves CV AUC 0.9934. And when submitted by itself achieves LB 0.064", "votes": 1 } ] }, { "id": 1257785, "postDate": "2021-03-31T05:11:18.987Z", "content": "Thanks for sharing ur great kernel! BTW, I have some questions here.
\nAlso, the calibrated confidence is genius. For score ordering, I used sum-based nms method that sums up scores of overlapping boxes that predicted by ensemble models.
\nThanks again !
", "rawMarkdown": "Thanks for sharing ur great kernel! BTW, I have some questions here.\n\n1. What`s the purpose of 15 class classifiers? \n2. What`s the meaning of cv on \"class14\"? Is it a model that distinguishes between find/nofind? \n\nAlso, the calibrated confidence is genius. For score ordering, I used sum-based nms method that sums up scores of overlapping boxes that predicted by ensemble models.\n\nThanks again !", "votes": 1, "replies": [ { "id": 1257792, "postDate": "2021-03-31T05:27:13.510Z", "content": "Thanks. Statistics say we only need to multiply the object detection confidence scores by the classifier probabililty \"no finding / finding\" scores to correct them. However through experimentation, we found we could increase CV mAP score more by also multiplying by the probability of class too.
\nSo for example if object detection says confidence for bbox of \"class 0\" is 0.78. Then we multiply 0.78 by probability that image has \"class 0\" and we multiply by probability that image has a \"finding\". We add exponents of 0.1 for each with formula
\n$$ score = base * prob(class)^{0.1} * prob(finding)^{0.1}$$
\nOur \"15 class classifier\" has 15 binary outputs and predicts the probability that a specific class is present. It was trained using the number of doctors who predict a class divided by 3 as ground truth.
\nWhen i say CV on \"class 14\", yes i mean the OOF AUC of binary classifier for \"no finding / finding\". The ensemble of 3 classifiers achieves CV OOF of AUC 0.9934
\nI like your idea your sum based NMS idea.
", "rawMarkdown": "Thanks. Statistics say we only need to multiply the object detection confidence scores by the classifier probabililty \"no finding / finding\" scores to correct them. However through experimentation, we found we could increase CV mAP score more by also multiplying by the probability of class too.\n\nSo for example if object detection says confidence for bbox of \"class 0\" is 0.78. Then we multiply 0.78 by probability that image has \"class 0\" and we multiply by probability that image has a \"finding\". We add exponents of 0.1 for each with formula\n\n$$ score = base * prob(class)^{0.1} * prob(finding)^{0.1}$$\n\nOur \"15 class classifier\" has 15 binary outputs and predicts the probability that a specific class is present. It was trained using the number of doctors who predict a class divided by 3 as ground truth.\n\nWhen i say CV on \"class 14\", yes i mean the OOF AUC of binary classifier for \"no finding / finding\". The ensemble of 3 classifiers achieves CV OOF of AUC 0.9934\n\nI like your idea your sum based NMS idea.", "votes": 2 }, { "id": 1257802, "postDate": "2021-03-31T05:36:04.047Z", "content": "Thanks for your details !
", "rawMarkdown": "Thanks for your details !", "votes": 1 } ] }, { "id": 1257780, "postDate": "2021-03-31T05:02:12.750Z", "content": "Congratulation on your final result. May I ask how many folds did you use for CV and did you stratify that or just randomly split the folds
", "rawMarkdown": "Congratulation on your final result. May I ask how many folds did you use for CV and did you stratify that or just randomly split the folds", "votes": 1, "replies": [ { "id": 1257784, "postDate": "2021-03-31T05:11:03.827Z", "content": "We had two validation schemes. The first was 10 stratified folds. We used this to make predictions on the test dataset. And we computed a CV mAP score on this.
\nNext, to compare different fold ensembling techniques, we had a second validation scheme. In our second validation scheme, we put fold 9 and 10 aside as a holdout validation set. The we trained an 8 fold model on folds 1-8 without adding fold 9 and 10 to train data. Lastly we used the 8 folds to predict folds 9 and 10. We compared different ways to ensemble the 8 folds like NMS versus WBF etc. We computed a validation mAP score on this.
", "rawMarkdown": "We had two validation schemes. The first was 10 stratified folds. We used this to make predictions on the test dataset. And we computed a CV mAP score on this.\n\nNext, to compare different fold ensembling techniques, we had a second validation scheme. In our second validation scheme, we put fold 9 and 10 aside as a holdout validation set. The we trained an 8 fold model on folds 1-8 without adding fold 9 and 10 to train data. Lastly we used the 8 folds to predict folds 9 and 10. We compared different ways to ensemble the 8 folds like NMS versus WBF etc. We computed a validation mAP score on this.", "votes": 1 }, { "id": 1257959, "postDate": "2021-03-31T08:35:42.620Z", "content": "The WBF has weights to denote the importances of different models when fusing boxes. Did you tune those, or use all weights=1 by default ?
", "rawMarkdown": "The WBF has weights to denote the importances of different models when fusing boxes. Did you tune those, or use all weights=1 by default ?", "votes": 1 }, { "id": 1258228, "postDate": "2021-03-31T13:14:58.137Z", "content": "We tried tuning the weights, but using weights=1 gave best CV score. First we applied NMS 0.4 to each fold separately. Next per model, we used WBF iou_thr=0.40, conf_type='max' to combine 10 model folds into 1 set of predictions. Next we used WBF iou_thr=0.40, conf_type='max' again to combine the 3 models into submission.csv
Congrats on 7th place ๐
", "rawMarkdown": "Congrats on 7th place ๐", "votes": 1 }, { "id": 1257719, "postDate": "2021-03-31T04:01:55.977Z", "content": "Congrats on 7th place @cdeotte and team
", "rawMarkdown": "Congrats on 7th place @cdeotte and team", "votes": 1 }, { "id": 1267101, "postDate": "2021-04-08T10:21:00.433Z", "content": "already had full pipelines to train VFNet, YoloV5, EfficientDet4 and more!
\nIt would be really beneficial for beginners if you shred your pipelines.
\nTo be honest, I tried to code a whole pipeline on my own but was never successful, hope to learn from your codes
\nit is better if its in PyTorch ; )
@cdeotte any reason for 0.1 as power?
Yes. It's a geometric mean. The formula p = a * b**0.1 * c **0.1 produces the same ranking ordering (i.e. same mAP) as a**0.84 * b**0.08 * c**0.08 = p**(5/6). This second expression is the geometric mean of numbers a, b, c with weights 0.84, 0.08, 0.08.
So we use 84% of the prediction from object detection model, 8% prediction from \"finding / no finding\" prediction, and 8% from \"specific class prediction\". These weights were found using local validation.
\nYou will also notice that Guanshuo Xu in his 6th place write-up used a geometric mean of a**0.84 * b**0.16. He used 84% object detection and 16% \"finding / no finding\" prediction. Most teams found this geometric mean to work best.
\n\n8% from \"specific class prediction\"
\n
But why did you need this class prediction? I suggest object detection models have a classifier inside.
\n(As far as I understand you had special 15-class classifier.)
\n\n\n8% prediction from \"finding / no finding\"
\n
This part is clear because you trained without empty images.
\nOr is it just empirical evidence that works better here?
", "rawMarkdown": "> 8% from \"specific class prediction\"\n\nBut why did you need this class prediction? I suggest object detection models have a classifier inside. \n\n(As far as I understand you had special 15-class classifier.)\n\n> 8% prediction from \"finding / no finding\" \n\nThis part is clear because you trained without empty images.\n\nOr is it just empirical evidence that works better here?\n", "votes": 1 }, { "id": 1261086, "postDate": "2021-04-02T17:19:54.883Z", "content": "Good point. I agree it shouldn't be needed. Bayesian statistics says only prediction from \"finding / no finding\" is needed when we train object detection on just positive images.
I think it helped for three reasons.
\nmAP metric doesn't want realistic probabilities. After predicting bbox, you can increase mAP by adding more bbox and using a \"class\" classifier helps find which additional bbox to add.But ultimately the decision was based on local validation. Using our local validation we tried both
\nThe latter increased CV and private LB by +0.002
", "rawMarkdown": "Good point. I agree it shouldn't be needed. Bayesian statistics says only `prediction from \"finding / no finding\"` is needed when we train object detection on just positive images.\n\nI think it helped for three reasons. \n* First, having the extra 14 outputs on our classifier probably made the output for \"finding/ no finding\" more accurate due to multi-task learning. \n* Second using 8% \"class\" and 8% \"finding\", is like ensembling two different diverse classifiers to achieve the goal of predicting \"finding\" better. \n* Third, the `mAP` metric doesn't want realistic probabilities. After predicting bbox, you can increase `mAP` by adding more bbox and using a \"class\" classifier helps find which additional bbox to add.\n\nBut ultimately the decision was based on local validation. Using our local validation we tried both \n* 16% \"finding\" classifier\n* 8% \"class\" ensembled with 8% \"finding\"\n\nThe latter increased CV and private LB by +0.002", "votes": 2 }, { "id": 1261190, "postDate": "2021-04-02T19:06:33.543Z", "content": "Wow, thank you for your detailed answer!
\nI think all your points are valid!
Congratz !
\n@cdeotte They should give you the Post-processing GM rank, I'm pretty sure that's the 5th gold you get with such ideas :)
", "rawMarkdown": "Congratz ! \n\n@cdeotte They should give you the Post-processing GM rank, I'm pretty sure that's the 5th gold you get with such ideas :)", "votes": 2, "replies": [ { "id": 1258618, "postDate": "2021-03-31T18:53:21.743Z", "content": "haha, true @theoviel
\nKaggle Post Process Grandmaster ๐
Congrats on the result. We also used the probability calibration technique in post-processing . Our calibrated confidence was calculated as: confidence score=(prob(class given finding=True)^2)โprob(finding=True)
\nWe missed the 15 class classifier part!! That was actually a very good idea !!
Congrats for your score too ! Yes the 15 class classifier had no big impact on LB but was very strong on CV so we trusted it
", "rawMarkdown": "Congrats for your score too ! Yes the 15 class classifier had no big impact on LB but was very strong on CV so we trusted it ", "votes": 1 }, { "id": 1258560, "postDate": "2021-03-31T17:55:50.197Z", "content": "Congratulations KS and team, well done!
\nJust to confirm, is your formula the \"output score from object detection squared\" multiplied by \"output score from classifier\"?
", "rawMarkdown": "Congratulations KS and team, well done!\n\nJust to confirm, is your formula the \"output score from object detection squared\" multiplied by \"output score from classifier\"?", "votes": 1 }, { "id": 1258568, "postDate": "2021-03-31T18:01:02.517Z", "content": "That should work good. That is the same as confidence score= prob(class given finding=True) โ prob(finding=True)**0.5 because both formulas keep scores in the same order. The formula here is just the square root of yours above.
For us, using confidence score= prob(class given finding=True) โ prob(finding=True)**0.2 worked nearly as well as our formula posted above which uses our 15 class classifier.
Did you try the formula
\nconfidence score=(prob(class given finding=True)^5)โprob(finding=True)
\nor the equivilent
\n confidence score= prob(class given finding=True) โ prob(finding=True)**0.2 ?
@cdeotte Yes, the formula is \"output score from object detection squared\" multiplied by \"output score from classifier\". I tried powers from 2 to 10 for prob ( class given finding=True ) on CV. I got the best results with power=2!! Results with power=2 and 3 were similar and it started dropping after power =4!!
", "rawMarkdown": "@cdeotte Yes, the formula is \"output score from object detection squared\" multiplied by \"output score from classifier\". I tried powers from 2 to 10 for prob ( class given finding=True ) on CV. I got the best results with power=2!! Results with power=2 and 3 were similar and it started dropping after power =4!!", "votes": 1 } ] }, { "id": 1257957, "postDate": "2021-03-31T08:34:40.187Z", "content": "Congrats!
", "rawMarkdown": "Congrats! ", "votes": 2 }, { "id": 1257902, "postDate": "2021-03-31T07:40:18.623Z", "content": "Congratulations @cdeotte , for detection, did you use 14 or 15 class training?
", "rawMarkdown": "Congratulations @cdeotte , for detection, did you use 14 or 15 class training?", "votes": 2, "replies": [ { "id": 1258028, "postDate": "2021-03-31T09:35:00.590Z", "content": "All our models are trained with 14 classes
", "rawMarkdown": "All our models are trained with 14 classes ", "votes": 2 } ] }, { "id": 1322642, "postDate": "2021-05-25T15:10:34.697Z", "content": "Hi, @cdeotte we are all fan of your post-processing techniques. Most of them are based on Bayesian methods. Can you share some good resources from where you have studied this.
\nThanks
Brilliant! From the melonoma classification, I have learned a lot from your discussion. thank you
\nI have a question why your team trained the models with 1024 but inferred with lower resolutions?
", "votes": 5, "replies": [] }, { "id": 1263240, "author_name": "David", "author_url": "", "post_date": "2021-04-05T08:30:04.340000", "content": "Hi @cdeotte , thank for sharing your post-processing approach. I have a question. You said the accuracy will increase if the boxes confidence score increase. In the Figure 2, how about if all confidence score all increases 0.1, so all the position from A-G still remains same as Figure 1 right, so is it make any change different from Figure 1?
", "votes": 1, "replies": [ { "id": 1263638, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-04-05T14:57:29.507000", "content": "No, then mAP will not increase. We increase mAP by rearranging the ranking order into a more accurate order just like the metric AUC.
Awesome post-processing approachโฆ as always :D
\nJust one question: do you think this approach is more robust and just generally works better than simply training with some empty images, and not using classifiers to calibrate confidence scores for each box?
", "votes": 1, "replies": [ { "id": 1259671, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-04-01T15:15:21.803000", "content": "Great question @ivanpan . Note that this post process is not the same as training object detection with all images.
\nBayesian statistics say we should use score = base * prob(finding) where base is the probability from object detection training on only \"finding\" images. I believe this formula would produce the same bbox confidence scores as training an object detection on all images.
The above formula is equivilent (i.e. same rank order, same mAP) as using score = base**0.5 * prob(finding)**0.5. This is the geometric mean between the two. Now, base is over zealous because it was trained on only positive images, and multiplying by prob(finding) makes the result realistic.
Our final formula is equivilent to score = base**0.84 * prob(finding)**0.16. This is the geometric average where we give more weight to over zealous bbox confidence scores which optimizes the loop holes of the mAP metric.
To clarify further, our formula is score = base * prob(finding)**0.1 * prob(class)**0.1 which is more or less score = base * prob(finding)**0.2 which is equivalent (same ranking ordering, same mAP) to score = base**0.84 * prob(finding)**0.16 if you raise the right side to the exponent 5/6.
Hm, interesting. I actually understand the math perfectly (+ saw your response about Xu's solution), but it doesn't seem intuitive at all :)
", "votes": 1, "replies": [] }, { "id": 1259801, "author_name": "Ivan Panshin", "author_url": "", "post_date": "2021-04-01T17:08:19.173000", "content": "Now that I think about it, it's actually more or less okay. Since your ending formula is pretty much the same as base**0.84 * prob(finding)**0.16, which is just geometric mean. But how to get 0.84/0.16? Grid search on [0, 1] with step 0.01 + check validation scores?
Yes, we found the formula using grid search on local validation.
\nYour question has me wondering how similar or different our post process is to training object detection on all images. As i said above, i think they are different and i think if i had an object detection model trained on all images, i could post process the confidence scores with a new formula not published anywhere that could increase it's mAP CV LB.
\nBecause the mAP metric rewards both precision and recall. And the best way to increase recall is to make more bbox on the images that we believe have findings (i.e. prob(finding)>0.5) and mAP does not penalize you for trying to find more bbox and being wrong.
Okay, it's official: my goal for 2021-2022 is to get into a team with you. I take models, you take post-processing ๐ช
", "votes": 0, "replies": [] } ] }, { "id": 1258818, "author_name": "Sidney Ng", "author_url": "", "post_date": "2021-03-31T23:08:24.803000", "content": "Very well explained @cdeotte ! Congrats once again !
", "votes": 1, "replies": [] }, { "id": 1258729, "author_name": "corochann", "author_url": "", "post_date": "2021-03-31T20:55:16.780000", "content": "Congrats! thanks for sharing insights about competition metric in detailed explanation.
\nI could not understand how to calibrate probability in \"Order2\" way, isn't it possible to calibrate only D & G class without knowing which is the true class??
", "votes": 1, "replies": [ { "id": 1258795, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T22:23:05.167000", "content": "Thanks. The letters are not classes. The entire picture is a single class. Each letter is one bbox. The numbers are the confidence scores assigned to a bbox. And the bboxes are all bboxes in your entire submission.csv file for the 1 pictured class. (This is a made up example with few bbox).
So given 1 class with its many bboxes in our submitted CSV file, if we can modify all the confidence scores in a smart way, then we can increase that 1 class's mAP.
\n\nif we can modify all the confidence scores in a smart way
\n
Thanks for reply, but I'm sorry that I still could not understand how to achieve this. Order does not affect to the metric, so just multiplying some value to all confidence does not change the metric.
\nThen, how you can change the order of confidence scores without knowing ground truth \"truth=0 or 1\"?
Congrats! I admire your post-processing very much.๐
", "votes": 1, "replies": [] }, { "id": 1258221, "author_name": "Sunghyun Jun", "author_url": "", "post_date": "2021-03-31T13:06:52.853000", "content": "Congrats and thanks for sharing your details! I have learned a lot from you. Thanks you.
", "votes": 1, "replies": [] }, { "id": 1257893, "author_name": "Tuan Ho Lai", "author_url": "", "post_date": "2021-03-31T07:30:21.377000", "content": "Congrats!! And thanks for the sharing in details, you always did a very fascinating job in post-process
", "votes": 1, "replies": [ { "id": 1258367, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T14:54:37.763000", "content": "Thanks Tuan
", "votes": 1, "replies": [] } ] }, { "id": 1257822, "author_name": "aqx", "author_url": "", "post_date": "2021-03-31T05:59:54.567000", "content": "Congratulations on the standing! It is people like you and your team whom really inspire beginner like myself to improve and become better. To have that intuition and experience to apply theories i never thought i could apply out of school ( Bayesian statistics ) ๐
", "votes": 1, "replies": [ { "id": 1257850, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T06:31:29.913000", "content": "Thanks. Congratulations on earning a bronze medal, top 10%
", "votes": 1, "replies": [] } ] }, { "id": 1257801, "author_name": "beluga", "author_url": "", "post_date": "2021-03-31T05:35:54.403000", "content": "Congrats for the good models and stable validation results! I like the postprocessing ideas!
\nI wonder if you tested your Class 14 predictions separately by just submitting the 1 pixel boxes what would be the LB AP score?
", "votes": 1, "replies": [ { "id": 1257817, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T05:53:28.913000", "content": "Thanks. We did not during the competition, but i just submitted now. Our ensembled classifier output for finding / no finding achieves CV AUC 0.9934. And when submitted by itself achieves LB 0.064
", "votes": 1, "replies": [] } ] }, { "id": 1257785, "author_name": "Wonho Song", "author_url": "", "post_date": "2021-03-31T05:11:18.987000", "content": "Thanks for sharing ur great kernel! BTW, I have some questions here.
\nAlso, the calibrated confidence is genius. For score ordering, I used sum-based nms method that sums up scores of overlapping boxes that predicted by ensemble models.
\nThanks again !
", "votes": 1, "replies": [ { "id": 1257792, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T05:27:13.510000", "content": "Thanks. Statistics say we only need to multiply the object detection confidence scores by the classifier probabililty \"no finding / finding\" scores to correct them. However through experimentation, we found we could increase CV mAP score more by also multiplying by the probability of class too.
\nSo for example if object detection says confidence for bbox of \"class 0\" is 0.78. Then we multiply 0.78 by probability that image has \"class 0\" and we multiply by probability that image has a \"finding\". We add exponents of 0.1 for each with formula
\n$$ score = base * prob(class)^{0.1} * prob(finding)^{0.1}$$
\nOur \"15 class classifier\" has 15 binary outputs and predicts the probability that a specific class is present. It was trained using the number of doctors who predict a class divided by 3 as ground truth.
\nWhen i say CV on \"class 14\", yes i mean the OOF AUC of binary classifier for \"no finding / finding\". The ensemble of 3 classifiers achieves CV OOF of AUC 0.9934
\nI like your idea your sum based NMS idea.
", "votes": 2, "replies": [] }, { "id": 1257802, "author_name": "Wonho Song", "author_url": "", "post_date": "2021-03-31T05:36:04.047000", "content": "Thanks for your details !
", "votes": 1, "replies": [] } ] }, { "id": 1257780, "author_name": "Liam Nguyen", "author_url": "", "post_date": "2021-03-31T05:02:12.750000", "content": "Congratulation on your final result. May I ask how many folds did you use for CV and did you stratify that or just randomly split the folds
", "votes": 1, "replies": [ { "id": 1257784, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T05:11:03.827000", "content": "We had two validation schemes. The first was 10 stratified folds. We used this to make predictions on the test dataset. And we computed a CV mAP score on this.
\nNext, to compare different fold ensembling techniques, we had a second validation scheme. In our second validation scheme, we put fold 9 and 10 aside as a holdout validation set. The we trained an 8 fold model on folds 1-8 without adding fold 9 and 10 to train data. Lastly we used the 8 folds to predict folds 9 and 10. We compared different ways to ensemble the 8 folds like NMS versus WBF etc. We computed a validation mAP score on this.
", "votes": 1, "replies": [] }, { "id": 1257959, "author_name": "Liam Nguyen", "author_url": "", "post_date": "2021-03-31T08:35:42.620000", "content": "The WBF has weights to denote the importances of different models when fusing boxes. Did you tune those, or use all weights=1 by default ?
", "votes": 1, "replies": [] }, { "id": 1258228, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T13:14:58.137000", "content": "We tried tuning the weights, but using weights=1 gave best CV score. First we applied NMS 0.4 to each fold separately. Next per model, we used WBF iou_thr=0.40, conf_type='max' to combine 10 model folds into 1 set of predictions. Next we used WBF iou_thr=0.40, conf_type='max' again to combine the 3 models into submission.csv
Congrats on 7th place ๐
", "votes": 1, "replies": [] }, { "id": 1257719, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-03-31T04:01:55.977000", "content": "Congrats on 7th place @cdeotte and team
", "votes": 1, "replies": [] }, { "id": 1267101, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-04-08T10:21:00.433000", "content": "already had full pipelines to train VFNet, YoloV5, EfficientDet4 and more!
\nIt would be really beneficial for beginners if you shred your pipelines.
\nTo be honest, I tried to code a whole pipeline on my own but was never successful, hope to learn from your codes
\nit is better if its in PyTorch ; )
@cdeotte any reason for 0.1 as power?
Yes. It's a geometric mean. The formula p = a * b**0.1 * c **0.1 produces the same ranking ordering (i.e. same mAP) as a**0.84 * b**0.08 * c**0.08 = p**(5/6). This second expression is the geometric mean of numbers a, b, c with weights 0.84, 0.08, 0.08.
So we use 84% of the prediction from object detection model, 8% prediction from \"finding / no finding\" prediction, and 8% from \"specific class prediction\". These weights were found using local validation.
\nYou will also notice that Guanshuo Xu in his 6th place write-up used a geometric mean of a**0.84 * b**0.16. He used 84% object detection and 16% \"finding / no finding\" prediction. Most teams found this geometric mean to work best.
\n\n8% from \"specific class prediction\"
\n
But why did you need this class prediction? I suggest object detection models have a classifier inside.
\n(As far as I understand you had special 15-class classifier.)
\n\n\n8% prediction from \"finding / no finding\"
\n
This part is clear because you trained without empty images.
\nOr is it just empirical evidence that works better here?
", "votes": 1, "replies": [] }, { "id": 1261086, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-04-02T17:19:54.883000", "content": "Good point. I agree it shouldn't be needed. Bayesian statistics says only prediction from \"finding / no finding\" is needed when we train object detection on just positive images.
I think it helped for three reasons.
\nmAP metric doesn't want realistic probabilities. After predicting bbox, you can increase mAP by adding more bbox and using a \"class\" classifier helps find which additional bbox to add.But ultimately the decision was based on local validation. Using our local validation we tried both
\nThe latter increased CV and private LB by +0.002
", "votes": 2, "replies": [] }, { "id": 1261190, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2021-04-02T19:06:33.543000", "content": "Wow, thank you for your detailed answer!
\nI think all your points are valid!
Congratz !
\n@cdeotte They should give you the Post-processing GM rank, I'm pretty sure that's the 5th gold you get with such ideas :)
", "votes": 2, "replies": [ { "id": 1258618, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T18:53:21.743000", "content": "haha, true @theoviel
\nKaggle Post Process Grandmaster ๐
Congrats on the result. We also used the probability calibration technique in post-processing . Our calibrated confidence was calculated as: confidence score=(prob(class given finding=True)^2)โprob(finding=True)
\nWe missed the 15 class classifier part!! That was actually a very good idea !!
Congrats for your score too ! Yes the 15 class classifier had no big impact on LB but was very strong on CV so we trusted it
", "votes": 1, "replies": [] }, { "id": 1258560, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T17:55:50.197000", "content": "Congratulations KS and team, well done!
\nJust to confirm, is your formula the \"output score from object detection squared\" multiplied by \"output score from classifier\"?
", "votes": 1, "replies": [] }, { "id": 1258568, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-03-31T18:01:02.517000", "content": "That should work good. That is the same as confidence score= prob(class given finding=True) โ prob(finding=True)**0.5 because both formulas keep scores in the same order. The formula here is just the square root of yours above.
For us, using confidence score= prob(class given finding=True) โ prob(finding=True)**0.2 worked nearly as well as our formula posted above which uses our 15 class classifier.
Did you try the formula
\nconfidence score=(prob(class given finding=True)^5)โprob(finding=True)
\nor the equivilent
\n confidence score= prob(class given finding=True) โ prob(finding=True)**0.2 ?
@cdeotte Yes, the formula is \"output score from object detection squared\" multiplied by \"output score from classifier\". I tried powers from 2 to 10 for prob ( class given finding=True ) on CV. I got the best results with power=2!! Results with power=2 and 3 were similar and it started dropping after power =4!!
", "votes": 1, "replies": [] } ] }, { "id": 1257957, "author_name": "CPMP", "author_url": "", "post_date": "2021-03-31T08:34:40.187000", "content": "Congrats!
", "votes": 2, "replies": [] }, { "id": 1257902, "author_name": "Awsaf", "author_url": "", "post_date": "2021-03-31T07:40:18.623000", "content": "Congratulations @cdeotte , for detection, did you use 14 or 15 class training?
", "votes": 2, "replies": [ { "id": 1258028, "author_name": "Matthieu Plantรฉ", "author_url": "", "post_date": "2021-03-31T09:35:00.590000", "content": "All our models are trained with 14 classes
", "votes": 2, "replies": [] } ] }, { "id": 1322642, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-05-25T15:10:34.697000", "content": "Hi, @cdeotte we are all fan of your post-processing techniques. Most of them are based on Bayesian methods. Can you share some good resources from where you have studied this.
\nThanks