\n
![\"\"](\"https://i.ibb.co/16cJfqh/165867751-1777430435773207-7505470057566631699-n.png\")
The metric for this competition is well-known, mean average precision(mAP). One of the features of this metric is No Penalty For Adding More Bbox @cdeotte. So, if we submit with lots of False Positives still we'll be able to get a good score. Check out the images below which are beyond comprehension. It was still able to get 0.26+ score on lb. So it really makes me think, does this metric even work?
\n
The metric for this competition is well-known, mean average precision(mAP). One of the features of this metric is No Penalty For Adding More Bbox @cdeotte. So, if we submit with lots of False Positives still we'll be able to get a good score. Check out the images below which are beyond comprehension. It was still able to get 0.26+ score on lb. So it really makes me think, does this metric even work?
\n
I think the potential problem with the metric is the following scenario. The purpose of a metric is to say \"one thing is better than another\", or \"two things are equal\". The following example have the same mAP. Do they have equal value to the host?
Both make 12 predictions and both have mAP = 0.50. (Note that mAP does take the area, but uses horizontal lines like pictured below). If our models are in production, we do not know which bbox are TP or FP, we just have predictions that are ordered by confidence scores.
If both of the below models are in production, and we select the 3 more confident bbox, then the model on the left is all TP and the model on the right is all FP. Are these models really equal? One could argue that the model on the left is better.
\nBoth have 50% area, but the one on the left has more area to the left. So one way to change the metric is to weight area on the left as more important than area on the right. (This would reward a model who top k bbox have more TP).
\n![\"\"](\"https://www.ccom.ucsd.edu/~cdeotte/Kaggle/compare-4-2.png\")
A little bit different metric was used in Pneumonia Detection Competition with same name mAP but there instead of Area Under the Curve, mAP is calculated using this formula,
\n
I was also curious why this metric was selected. Same thoughts - huge number of bboxes with conf scores < 0.001 which make the predictions incomprehensible, yet increase the mAP score.
\nI was thinking maybe its because during medical diagnostics you would rather have a lot of predictions which include correct abnormality (TP) + lots of FPs, rather than a limited number of predictions where the TP might be missed out.
\nBut this is just a guess - I am new to object detection and haven't worked with other metrics yet, so maybe other metrics have different limitations.
Yes you're right sometimes TP is more important than FP but with that many FP I'm not sure how health workers will treat the patient.
Hi @cdeotte, I think your comment somehow got deleted, I'm talking about this scenario. If we take top k then those low-confident boxes will be removed and it'll reduce the score. ![\"\"](\"https://www.ccom.ucsd.edu/~cdeotte/Kaggle/map3.png\")
I think the potential problem with the metric is the following scenario. The purpose of a metric is to say \"one thing is better than another\", or \"two things are equal\". The following example have the same mAP. Do they have equal value to the host?
Both make 12 predictions and both have mAP = 0.50. (Note that mAP does take the area, but uses horizontal lines like pictured below). If our models are in production, we do not know which bbox are TP or FP, we just have predictions that are ordered by confidence scores.
If both of the below models are in production, and we select the 3 more confident bbox, then the model on the left is all TP and the model on the right is all FP. Are these models really equal? One could argue that the model on the left is better.
\nBoth have 50% area, but the one on the left has more area to the left. So one way to change the metric is to weight area on the left as more important than area on the right. (This would reward a model who top k bbox have more TP).
\n
A little bit different metric was used in Pneumonia Detection Competition with same name mAP but there instead of Area Under the Curve, mAP is calculated using this formula,
\n
I was also curious why this metric was selected. Same thoughts - huge number of bboxes with conf scores < 0.001 which make the predictions incomprehensible, yet increase the mAP score.
\nI was thinking maybe its because during medical diagnostics you would rather have a lot of predictions which include correct abnormality (TP) + lots of FPs, rather than a limited number of predictions where the TP might be missed out.
\nBut this is just a guess - I am new to object detection and haven't worked with other metrics yet, so maybe other metrics have different limitations.
Yes you're right sometimes TP is more important than FP but with that many FP I'm not sure how health workers will treat the patient.
Hi @cdeotte, I think your comment somehow got deleted, I'm talking about this scenario. If we take top k then those low-confident boxes will be removed and it'll reduce the score.