Hi there,
\nnow we know 10% of total 3000 test data are public, rest 90% are private.
\nwe can estimate host will prepare relatively more abnormal images for test, but if test distribution of each abnormalities are same as train/public test/private test, N of each abnormalities in test data
\ncan be estimated as
\npublic : [61, 3, 9, 46, 7, 7, 12, 26, 16, 22, 20, 39, 1, 32],
\nprivate : [552, 33, 81, 413, 63, 69, 110, 237, 148, 204, 185, 356, 17, 291]
\nthis may cause huge shakeup for last leaderboard. how do you think?
Hi there,
\nnow we know 10% of total 3000 test data are public, rest 90% are private.
\nwe can estimate host will prepare relatively more abnormal images for test, but if test distribution of each abnormalities are same as train/public test/private test, N of each abnormalities in test data
\ncan be estimated as
\npublic : [61, 3, 9, 46, 7, 7, 12, 26, 16, 22, 20, 39, 1, 32],
\nprivate : [552, 33, 81, 413, 63, 69, 110, 237, 148, 204, 185, 356, 17, 291]
\nthis may cause huge shakeup for last leaderboard. how do you think?
I think you're right. Even if you re-run your model for several times, the LB score will be quite different with little init randomization. And considering the annotations from train/test set are also different, I'm still having a hard time trying to find a reliable metric to select model.
", "rawMarkdown": "I think you're right. Even if you re-run your model for several times, the LB score will be quite different with little init randomization. And considering the annotations from train/test set are also different, I'm still having a hard time trying to find a reliable metric to select model.", "votes": 4, "replies": [ { "id": 1210227, "postDate": "2021-02-19T09:24:36.240Z", "content": "@issacc I got lb range 12.0-18.0 map when re-running same model… I am frustrated, what am i doing wrong lol?
", "rawMarkdown": "@issacc I got lb range 12.0-18.0 map when re-running same model... I am frustrated, what am i doing wrong lol?", "votes": 2 }, { "id": 1211677, "postDate": "2021-02-20T12:44:00.320Z", "content": "Thanks for sharing ideas, we wanna get higher public/private lb, but when we seek upper public lb score, we have more risk for shake at private lb for espessially in this competition. dilemma🤔
", "rawMarkdown": "Thanks for sharing ideas, we wanna get higher public/private lb, but when we seek upper public lb score, we have more risk for shake at private lb for espessially in this competition. dilemma🤔", "votes": 1 } ] }, { "id": 1215290, "postDate": "2021-02-23T14:04:01.527Z", "content": "Agree!!
\nwhich makes is more important to trust your CV and models and not get swayed by the Public LB
Hello , is this game only need to submit a table?
", "rawMarkdown": "Hello , is this game only need to submit a table?", "votes": 1 }, { "id": 1207011, "postDate": "2021-02-17T17:18:17.643Z", "content": "I was just thinking: Looking at your frequency estimates, model selection based on public LB results restricted to the more frequent classes may be an option? These might be a bit more reliable than the very rare classes? On the other hand, different models may behaviour differently on rare vs. frequent classes.. But this I guess can only be assessed looking at the respective model and setup individualistically..
", "rawMarkdown": "I was just thinking: Looking at your frequency estimates, model selection based on public LB results restricted to the more frequent classes may be an option? These might be a bit more reliable than the very rare classes? On the other hand, different models may behaviour differently on rare vs. frequent classes.. But this I guess can only be assessed looking at the respective model and setup individualistically..", "votes": 1, "replies": [ { "id": 1211668, "postDate": "2021-02-20T12:29:10.643Z", "content": "I guess we can roughly refar to public LB results, for ex, if each models are single, public LB 0.25 vs 0.1, will be the same results as Private LB, but the more we start to add new process/system to test data,
\n(for ex, re-thinking bbox(nms, nmw, wbf, etc…)), you have more possibility of overfitting to Public LB.
\n(for my experience, Hold-out cv 0.25 vs Public LB 0.16 for simple model outputs, hold-out cv 0.35 vs Public LB 0.12 for additional process.)
Yes, this aggrevates the issue.. In my experiment I included the 2-class predictions and NMS so that might be an issue.. on the other hand, I still think that a single model may accidentally fit very well let's say the few cases of Atelectasis in the public LB set while an other model might not. At the same time the latter model could show a better performance on the private LB set where more cases of Atelectasis are included.
", "rawMarkdown": "Yes, this aggrevates the issue.. In my experiment I included the 2-class predictions and NMS so that might be an issue.. on the other hand, I still think that a single model may accidentally fit very well let's say the few cases of Atelectasis in the public LB set while an other model might not. At the same time the latter model could show a better performance on the private LB set where more cases of Atelectasis are included.", "votes": 2 } ] }, { "id": 1199078, "postDate": "2021-02-13T14:05:17.437Z", "content": "All 3000 test images are in public. isn't it?
\nam new to the competition so can you please tell me whether the leaderboard test data will be different from the available test data.
", "rawMarkdown": "All 3000 test images are in public. isn't it? \n\nam new to the competition so can you please tell me whether the leaderboard test data will be different from the available test data.", "votes": 1, "replies": [ { "id": 1199169, "postDate": "2021-02-13T15:02:18.163Z", "content": "They're all \"public\" as in you can view them, but only 300 are scored on the public LB. That's what he means.
", "rawMarkdown": "They're all \"public\" as in you can view them, but only 300 are scored on the public LB. That's what he means.", "votes": 3 }, { "id": 1200851, "postDate": "2021-02-15T03:47:38.020Z", "content": "If I understand correctly, 3000 test images provided by the host include 300 for LB and 2700 remains for private LB?
", "rawMarkdown": "If I understand correctly, 3000 test images provided by the host include 300 for LB and 2700 remains for private LB?", "votes": 2 }, { "id": 1200853, "postDate": "2021-02-15T03:50:42.050Z", "content": "Sure. right
", "rawMarkdown": "Sure. right", "votes": 2 } ] }, { "id": 1248038, "postDate": "2021-03-22T09:32:41.150Z", "content": "I think there would be shake up.
\nAll previous X-ray competition had a great shake up. So I expect the same here.
\nBut ….
\nto trust on CV is quite hard in this competition. Since
\nCV doesn't correlate nor proportional to LB Score
Yes, since my hold-out cv/lb score became mess, I stopped submitting,
\nthere are some factors of shake, public/private test label distribution difference, stage2 manual
\nlabeling from some radiologists, noisy training dataset of training set…etc
\nI guess last private lb top will be 0.2〜0.3, how do you think?
I think the same. but still i could imagine sometime our CV score might be the final private LB score
", "rawMarkdown": "I think the same. but still i could imagine sometime our CV score might be the final private LB score" }, { "id": 1248243, "postDate": "2021-03-22T13:13:17.363Z", "content": "@morizin what is your best cv so far?
", "rawMarkdown": "@morizin what is your best cv so far? " }, { "id": 1248246, "postDate": "2021-03-22T13:16:29.590Z", "content": "The final CV is being calculating now
", "rawMarkdown": "The final CV is being calculating now" } ] }, { "id": 1249225, "postDate": "2021-03-23T07:23:17.903Z", "content": "Shake-up is obvious in this competition base on the antecedent of previous similar ones.
\nThe LB will bring forth this trend. Based on discussion trends a lot of participants expect the shake-up but do not know what structure it will be introduced.
I think you're right. Even if you re-run your model for several times, the LB score will be quite different with little init randomization. And considering the annotations from train/test set are also different, I'm still having a hard time trying to find a reliable metric to select model.
", "votes": 4, "replies": [ { "id": 1210227, "author_name": "Gleb Anferov", "author_url": "", "post_date": "2021-02-19T09:24:36.240000", "content": "@issacc I got lb range 12.0-18.0 map when re-running same model… I am frustrated, what am i doing wrong lol?
", "votes": 2, "replies": [] }, { "id": 1211677, "author_name": "Ryunosuke Ishizaki", "author_url": "", "post_date": "2021-02-20T12:44:00.320000", "content": "Thanks for sharing ideas, we wanna get higher public/private lb, but when we seek upper public lb score, we have more risk for shake at private lb for espessially in this competition. dilemma🤔
", "votes": 1, "replies": [] } ] }, { "id": 1215290, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-02-23T14:04:01.527000", "content": "Agree!!
\nwhich makes is more important to trust your CV and models and not get swayed by the Public LB
Hello , is this game only need to submit a table?
", "votes": 1, "replies": [] }, { "id": 1207011, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-02-17T17:18:17.643000", "content": "I was just thinking: Looking at your frequency estimates, model selection based on public LB results restricted to the more frequent classes may be an option? These might be a bit more reliable than the very rare classes? On the other hand, different models may behaviour differently on rare vs. frequent classes.. But this I guess can only be assessed looking at the respective model and setup individualistically..
", "votes": 1, "replies": [ { "id": 1211668, "author_name": "Ryunosuke Ishizaki", "author_url": "", "post_date": "2021-02-20T12:29:10.643000", "content": "I guess we can roughly refar to public LB results, for ex, if each models are single, public LB 0.25 vs 0.1, will be the same results as Private LB, but the more we start to add new process/system to test data,
\n(for ex, re-thinking bbox(nms, nmw, wbf, etc…)), you have more possibility of overfitting to Public LB.
\n(for my experience, Hold-out cv 0.25 vs Public LB 0.16 for simple model outputs, hold-out cv 0.35 vs Public LB 0.12 for additional process.)
Yes, this aggrevates the issue.. In my experiment I included the 2-class predictions and NMS so that might be an issue.. on the other hand, I still think that a single model may accidentally fit very well let's say the few cases of Atelectasis in the public LB set while an other model might not. At the same time the latter model could show a better performance on the private LB set where more cases of Atelectasis are included.
", "votes": 2, "replies": [] } ] }, { "id": 1199078, "author_name": " Sachin", "author_url": "", "post_date": "2021-02-13T14:05:17.437000", "content": "All 3000 test images are in public. isn't it?
\nam new to the competition so can you please tell me whether the leaderboard test data will be different from the available test data.
", "votes": 1, "replies": [ { "id": 1199169, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2021-02-13T15:02:18.163000", "content": "They're all \"public\" as in you can view them, but only 300 are scored on the public LB. That's what he means.
", "votes": 3, "replies": [] }, { "id": 1200851, "author_name": "Phat Tran", "author_url": "", "post_date": "2021-02-15T03:47:38.020000", "content": "If I understand correctly, 3000 test images provided by the host include 300 for LB and 2700 remains for private LB?
", "votes": 2, "replies": [] }, { "id": 1200853, "author_name": "Ryunosuke Ishizaki", "author_url": "", "post_date": "2021-02-15T03:50:42.050000", "content": "Sure. right
", "votes": 2, "replies": [] } ] }, { "id": 1248038, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-03-22T09:32:41.150000", "content": "I think there would be shake up.
\nAll previous X-ray competition had a great shake up. So I expect the same here.
\nBut ….
\nto trust on CV is quite hard in this competition. Since
\nCV doesn't correlate nor proportional to LB Score
Yes, since my hold-out cv/lb score became mess, I stopped submitting,
\nthere are some factors of shake, public/private test label distribution difference, stage2 manual
\nlabeling from some radiologists, noisy training dataset of training set…etc
\nI guess last private lb top will be 0.2〜0.3, how do you think?
I think the same. but still i could imagine sometime our CV score might be the final private LB score
", "votes": 0, "replies": [] }, { "id": 1248243, "author_name": "Awsaf", "author_url": "", "post_date": "2021-03-22T13:13:17.363000", "content": "@morizin what is your best cv so far?
", "votes": 0, "replies": [] }, { "id": 1248246, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-03-22T13:16:29.590000", "content": "The final CV is being calculating now
", "votes": 0, "replies": [] } ] }, { "id": 1249225, "author_name": "Olusesi Adebisi", "author_url": "", "post_date": "2021-03-23T07:23:17.903000", "content": "Shake-up is obvious in this competition base on the antecedent of previous similar ones.
\nThe LB will bring forth this trend. Based on discussion trends a lot of participants expect the shake-up but do not know what structure it will be introduced.