First time I'm seeing that nms performs better on both CV and LB.
\nI guess noisy labels has something to do with this, but it is surprising and drives me crazy.
\nMaybe soft-nms does better?
| \n | CV | \nLB | \n
|---|---|---|
| nms | \n0.36 | \n0.22 | \n
| WBF | \n0.33 | \n0.19 | \n
First time I'm seeing that nms performs better on both CV and LB.
\nI guess noisy labels has something to do with this, but it is surprising and drives me crazy.
\nMaybe soft-nms does better?
| \n | CV | \nLB | \n
|---|---|---|
| nms | \n0.36 | \n0.22 | \n
| WBF | \n0.33 | \n0.19 | \n
Thanks for sharing interesting topic!
\nI also considered and I think that NMS is more similar to the process what is done to create test dataset.
\n2 reviewer selects better bbox (done in NMS), instead of creating \"averaged bbox\" (done in WBF).
I also wrote some idea to weight NMS based on rad_id in the discussion: https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/219221
oh, somehow I thought the reviewer creates an averaged box. I should read the original paper..
\nYour discussion was interesting too, thanks for raising the topic.
I got the same results too. But WBF improves my CV score by about 2AP. However, no matter how I tune the 'iou_thr', I can not obtain a higher score on LB, by using WBF.
", "rawMarkdown": "I got the same results too. But WBF improves my CV score by about 2AP. However, no matter how I tune the 'iou_thr', I can not obtain a higher score on LB, by using WBF.", "votes": 1, "replies": [ { "id": 1175791, "postDate": "2021-01-29T10:56:56.877Z", "content": "given the metric is only iou@0.4, its pretty odd to see such changes with the nms/wbf..
", "rawMarkdown": "given the metric is only iou@0.4, its pretty odd to see such changes with the nms/wbf.." }, { "id": 1175810, "postDate": "2021-01-29T11:06:58.027Z", "content": "How do you implement the wbf? Did you use the method implemented in \"ensemble-boxes\"?
", "rawMarkdown": "How do you implement the wbf? Did you use the method implemented in \"ensemble-boxes\"?" }, { "id": 1175897, "postDate": "2021-01-29T11:55:48.083Z", "content": "Yes, just used the one in zfturbo's repo.
", "rawMarkdown": "Yes, just used the one in zfturbo's repo." }, { "id": 1178968, "postDate": "2021-01-31T08:25:26.980Z", "content": "Sorry for the late reply. I have used the one in zfturbo's repo too. Since the performance is worse than
\n'nms' on PB. So I re-implemented one by myself, the results are almost the same.
Even thats my case i got a bad result with NMS also with Soft NMS
", "rawMarkdown": "Even thats my case i got a bad result with NMS also with Soft NMS" }, { "id": 1179070, "postDate": "2021-01-31T09:44:53.180Z", "content": "@morizin Then, you got better results with WBF?
", "rawMarkdown": "@morizin Then, you got better results with WBF?", "votes": 1 } ] }, { "id": 1193901, "postDate": "2021-02-10T00:22:53.577Z", "content": "@zehuigong Why do you have to tune 'iou_thr'? isn't it supposed to be fixed at 0.4 ?
", "rawMarkdown": "@zehuigong Why do you have to tune 'iou_thr'? isn't it supposed to be fixed at 0.4 ?", "replies": [ { "id": 1193960, "postDate": "2021-02-10T02:22:00.390Z", "content": "You can tune the iou thresholds of the nms.
\nIt will change how to merge the overlapping boxes.
The mAP evaluation metric is fixed at iou=0.4
", "rawMarkdown": "You can tune the iou thresholds of the nms.\nIt will change how to merge the overlapping boxes.\n\nThe mAP evaluation metric is fixed at iou=0.4" } ] }, { "id": 1175539, "postDate": "2021-01-29T08:13:16.737Z", "content": "\n\nMaybe soft-nms does better?
\n
Did you use Soft-NMS or NMS?
", "rawMarkdown": "> Maybe soft-nms does better?\n\nDid you use Soft-NMS or NMS?", "replies": [ { "id": 1175790, "postDate": "2021-01-29T10:55:59.487Z", "content": "i used normal nms
", "rawMarkdown": "i used normal nms", "votes": 1 } ] } ], "comments": [ { "id": 1199596, "author_name": "corochann", "author_url": "", "post_date": "2021-02-14T01:33:59.743000", "content": "Thanks for sharing interesting topic!
\nI also considered and I think that NMS is more similar to the process what is done to create test dataset.
\n2 reviewer selects better bbox (done in NMS), instead of creating \"averaged bbox\" (done in WBF).
I also wrote some idea to weight NMS based on rad_id in the discussion: https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/219221
oh, somehow I thought the reviewer creates an averaged box. I should read the original paper..
\nYour discussion was interesting too, thanks for raising the topic.
I got the same results too. But WBF improves my CV score by about 2AP. However, no matter how I tune the 'iou_thr', I can not obtain a higher score on LB, by using WBF.
", "votes": 1, "replies": [ { "id": 1175791, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-29T10:56:56.877000", "content": "given the metric is only iou@0.4, its pretty odd to see such changes with the nms/wbf..
", "votes": 0, "replies": [] }, { "id": 1175810, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-29T11:06:58.027000", "content": "How do you implement the wbf? Did you use the method implemented in \"ensemble-boxes\"?
", "votes": 0, "replies": [] }, { "id": 1175897, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-29T11:55:48.083000", "content": "Yes, just used the one in zfturbo's repo.
", "votes": 0, "replies": [] }, { "id": 1178968, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-31T08:25:26.980000", "content": "Sorry for the late reply. I have used the one in zfturbo's repo too. Since the performance is worse than
\n'nms' on PB. So I re-implemented one by myself, the results are almost the same.
Even thats my case i got a bad result with NMS also with Soft NMS
", "votes": 0, "replies": [] }, { "id": 1179070, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-31T09:44:53.180000", "content": "@morizin Then, you got better results with WBF?
", "votes": 1, "replies": [] } ] }, { "id": 1193901, "author_name": "sourabhsc", "author_url": "", "post_date": "2021-02-10T00:22:53.577000", "content": "@zehuigong Why do you have to tune 'iou_thr'? isn't it supposed to be fixed at 0.4 ?
", "votes": 0, "replies": [ { "id": 1193960, "author_name": "arutema47", "author_url": "", "post_date": "2021-02-10T02:22:00.390000", "content": "You can tune the iou thresholds of the nms.
\nIt will change how to merge the overlapping boxes.
The mAP evaluation metric is fixed at iou=0.4
", "votes": 0, "replies": [] } ] }, { "id": 1175539, "author_name": "Phat Tran", "author_url": "", "post_date": "2021-01-29T08:13:16.737000", "content": "\n\nMaybe soft-nms does better?
\n
Did you use Soft-NMS or NMS?
", "votes": 0, "replies": [ { "id": 1175790, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-29T10:55:59.487000", "content": "i used normal nms
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1173724": "First time I'm seeing that nms performs better on both CV and LB.\nI guess noisy labels has something to do with this, but it is surprising and drives me crazy.\nMaybe soft-nms does better?\n\n| | CV | LB |\n| --- | --- | --- |\n| nms | 0.36 | 0.22 |\n|WBF | 0.33 | 0.19 |\n", "1199596": "Thanks for sharing interesting topic!\nI also considered and I think that NMS is more similar to the process what is done to create test dataset.\n**2 reviewer selects better bbox (done in NMS), instead of creating \"averaged bbox\" (done in WBF)**.\n\nI also wrote some idea to weight NMS based on `rad_id` in the discussion: https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/219221", "1173971": "I got the same results too. But WBF improves my CV score by about 2AP. However, no matter how I tune the 'iou_thr', I can not obtain a higher score on LB, by using WBF.", "1193901": "@zehuigong Why do you have to tune 'iou_thr'? isn't it supposed to be fixed at 0.4 ?", "1175539": "> Maybe soft-nms does better?\n\nDid you use Soft-NMS or NMS?" } }