{ "id": 214284, "title": "The correlation between CV/Public score", "url": "/competitions/vinbigdata-chest-xray-abnormalities-detection/discussion/214284", "author_name": "ZehuiGong", "post_date": "2021-01-26T03:14:30.661000", "votes": 18, "comment_count": 28, "views": 0, "content": "

Hi, everyone. I found an issue here. My scores on CV vs. LB can not correspond to each other. Below is an example:
\nCV: 0.332 --> LB: 0.217
\nCV: 0.350 --> LB: 0.173
\nCV: 0.331 -->LB: 0.179
\nI am confused about the correlation between CV and LB. How are your scores on CV vs. LB? Posting mine below.

", "messages": [ { "id": 1170154, "postDate": "2021-01-26T03:14:30.663Z", "content": "

Hi, everyone. I found an issue here. My scores on CV vs. LB can not correspond to each other. Below is an example:
\nCV: 0.332 --> LB: 0.217
\nCV: 0.350 --> LB: 0.173
\nCV: 0.331 -->LB: 0.179
\nI am confused about the correlation between CV and LB. How are your scores on CV vs. LB? Posting mine below.

", "rawMarkdown": "Hi, everyone. I found an issue here. My scores on CV vs. LB can not correspond to each other. Below is an example:\nCV: 0.332 --> LB: 0.217\nCV: 0.350 --> LB: 0.173\nCV: 0.331 -->LB: 0.179\nI am confused about the correlation between CV and LB. How are your scores on CV vs. LB? Posting mine below.", "votes": 18 }, { "id": 1192229, "postDate": "2021-02-09T03:49:20.980Z", "content": "

Hi, Are your CV include \"No findings\" class and have you apply \"2 class filtering\" on your LB?

", "rawMarkdown": "Hi, Are your CV include \"No findings\" class and have you apply \"2 class filtering\" on your LB?", "replies": [ { "id": 1193069, "postDate": "2021-02-09T12:45:48.143Z", "content": "

I found that there is no difference on CV, with or without \"No findings\". Yes, I have applied '2-class filtering'.

", "rawMarkdown": "I found that there is no difference on CV, with or without \"No findings\". Yes, I have applied '2-class filtering'.", "votes": 1 }, { "id": 1193141, "postDate": "2021-02-09T13:38:23.263Z", "content": "

I think it's because of noise in the annotation. As discussed here.

\n

My CV is ~0.27 and got only 0.102 PB (without 2-class filtering)

", "rawMarkdown": "I think it's because of noise in the annotation. As discussed [here](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/212859).\n\nMy CV is ~0.27 and got only 0.102 PB (without 2-class filtering)" } ] }, { "id": 1188203, "postDate": "2021-02-06T04:15:41.630Z", "content": "
\n

CV: 0.332 --> LB: 0.217
\n CV: 0.350 --> LB: 0.173
\n CV: 0.331 -->LB: 0.179

\n
\n

Are these single models? @zehuigong

", "rawMarkdown": "> CV: 0.332 --> LB: 0.217\nCV: 0.350 --> LB: 0.173\nCV: 0.331 -->LB: 0.179\n\nAre these single models? @zehuigong ", "replies": [ { "id": 1190141, "postDate": "2021-02-07T14:09:17.947Z", "content": "

Yes, these are the result of a single model and single fold.

", "rawMarkdown": "Yes, these are the result of a single model and single fold." } ] }, { "id": 1183045, "postDate": "2021-02-02T17:58:05.633Z", "content": "

The host said that , the test set is also annotated by 3 radiologists but these boxes were unified to one single box by 2 professional radiologists.
\nEither the 3 radiologists or 2 radiologists would be different. Maybe they are different in their annotations.

", "rawMarkdown": "The host said that , the test set is also annotated by 3 radiologists but these boxes were unified to one single box by 2 professional radiologists.\nEither the 3 radiologists or 2 radiologists would be different. Maybe they are different in their annotations.", "replies": [ { "id": 1190154, "postDate": "2021-02-07T14:15:43.017Z", "content": "

Sorry for the late reply. As far as I know, the disagreement will be further judged by the radiologists and finally made a consensus. In my opinion, the difference in annotation distribution between train/test sets makes this competition a more challenging one.

", "rawMarkdown": "Sorry for the late reply. As far as I know, the disagreement will be further judged by the radiologists and finally made a consensus. In my opinion, the difference in annotation distribution between train/test sets makes this competition a more challenging one." } ] }, { "id": 1181146, "postDate": "2021-02-01T17:20:28.553Z", "content": "

I guess we're about to see a huge shake up

", "rawMarkdown": "I guess we're about to see a huge shake up", "replies": [ { "id": 1181643, "postDate": "2021-02-02T03:10:00.500Z", "content": "

Given the results that we have got now. I feel the same too.

", "rawMarkdown": "Given the results that we have got now. I feel the same too. " } ] }, { "id": 1174174, "postDate": "2021-01-28T10:18:08.937Z", "content": "

i have the same problem. CV got 0.36 but PL so bad.

", "rawMarkdown": "i have the same problem. CV got 0.36 but PL so bad. ", "replies": [ { "id": 1174200, "postDate": "2021-01-28T10:36:38.760Z", "content": "

Maybe we can trust our CV score and pay less attention to the PL.

", "rawMarkdown": "Maybe we can trust our CV score and pay less attention to the PL." } ] }, { "id": 1170491, "postDate": "2021-01-26T08:47:09.157Z", "content": "

One of the thing we can understand from this is test set is almost different from the training set . we cannot find a reasonable score on LB as we get in CV

", "rawMarkdown": "One of the thing we can understand from this is test set is almost different from the training set . we cannot find a reasonable score on LB as we get in CV", "replies": [ { "id": 1170506, "postDate": "2021-01-26T08:56:14.710Z", "content": "

Yes, if this is true, it is hard to improve our baseline model, because we do not know which score we could trust, CV or LB. May I ask does your improvement on CV also results in the improvement on LB?

", "rawMarkdown": "Yes, if this is true, it is hard to improve our baseline model, because we do not know which score we could trust, CV or LB. May I ask does your improvement on CV also results in the improvement on LB?" }, { "id": 1170591, "postDate": "2021-01-26T09:56:36.630Z", "content": "

Yes My CV is almost 0.5 for the score of 0.318. We cant trust PublicLB because it is doing only in 300 images

", "rawMarkdown": "Yes My CV is almost 0.5 for the score of 0.318. We cant trust PublicLB because it is doing only in 300 images", "votes": 1 }, { "id": 1171040, "postDate": "2021-01-26T15:42:57.837Z", "content": "

@morizin, may I ask how you split your whole training set into the cross-validation set?

", "rawMarkdown": "@morizin, may I ask how you split your whole training set into the cross-validation set?" }, { "id": 1173981, "postDate": "2021-01-28T08:02:13.953Z", "content": "

I have CV-LB disparencies too.
\nGuess it's a trust-CV competition given the small size of the public set.

", "rawMarkdown": "I have CV-LB disparencies too. \nGuess it's a trust-CV competition given the small size of the public set.", "votes": 2 }, { "id": 1174092, "postDate": "2021-01-28T09:05:58.310Z", "content": "

@zehuigong I have done StratifiedKFold with labels

", "rawMarkdown": "@zehuigong I have done StratifiedKFold with labels" }, { "id": 1174204, "postDate": "2021-01-28T10:40:12.593Z", "content": "

Got it, thanks for your sharing.

", "rawMarkdown": "Got it, thanks for your sharing." }, { "id": 1174231, "postDate": "2021-01-28T11:00:40.483Z", "content": "

but why do you need mine it doesn't correlate with LB

", "rawMarkdown": "but why do you need mine it doesn't correlate with LB" }, { "id": 1174268, "postDate": "2021-01-28T11:18:39.610Z", "content": "

Now, I think that many of us have encountered this issue, the CV-LB discrepancies.

", "rawMarkdown": "Now, I think that many of us have encountered this issue, the CV-LB discrepancies. " }, { "id": 1181075, "postDate": "2021-02-01T16:26:44.103Z", "content": "

Are you measuring CV mAP against the raw ground truth? Or against NMS/WBF-transformed boxes?

", "rawMarkdown": "Are you measuring CV mAP against the raw ground truth? Or against NMS/WBF-transformed boxes?" }, { "id": 1181085, "postDate": "2021-02-01T16:32:11.173Z", "content": "

I checked both with and without never correlates

", "rawMarkdown": "I checked both with and without never correlates" } ] }, { "id": 1174651, "postDate": "2021-01-28T16:22:11.443Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1175298, "postDate": "2021-01-29T04:32:06.910Z", "content": "

Have you tried different cross-validation set divisions, i.e., generating different 5 fold cross-validation?

", "rawMarkdown": "Have you tried different cross-validation set divisions, i.e., generating different 5 fold cross-validation?" }, { "id": 1181525, "postDate": "2021-02-02T00:38:42.497Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1181646, "postDate": "2021-02-02T03:12:23.610Z", "content": "

Of all the divisions you make, none of them show a correlation between CV and LB score?

", "rawMarkdown": "Of all the divisions you make, none of them show a correlation between CV and LB score?" }, { "id": 1182975, "postDate": "2021-02-02T17:04:53.377Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1182987, "postDate": "2021-02-02T17:11:27.850Z", "content": "

May I ask what kind of detection architecture(algorithm) are you experimenting with?

", "rawMarkdown": "May I ask what kind of detection architecture(algorithm) are you experimenting with?" } ] } ], "comments": [ { "id": 1192229, "author_name": "Phat Tran", "author_url": "", "post_date": "2021-02-09T03:49:20.980000", "content": "

Hi, Are your CV include \"No findings\" class and have you apply \"2 class filtering\" on your LB?

", "votes": 0, "replies": [ { "id": 1193069, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-02-09T12:45:48.143000", "content": "

I found that there is no difference on CV, with or without \"No findings\". Yes, I have applied '2-class filtering'.

", "votes": 1, "replies": [] }, { "id": 1193141, "author_name": "Phat Tran", "author_url": "", "post_date": "2021-02-09T13:38:23.263000", "content": "

I think it's because of noise in the annotation. As discussed here.

\n

My CV is ~0.27 and got only 0.102 PB (without 2-class filtering)

", "votes": 0, "replies": [] } ] }, { "id": 1188203, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-02-06T04:15:41.630000", "content": "
\n

CV: 0.332 --> LB: 0.217
\n CV: 0.350 --> LB: 0.173
\n CV: 0.331 -->LB: 0.179

\n
\n

Are these single models? @zehuigong

", "votes": 0, "replies": [ { "id": 1190141, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-02-07T14:09:17.947000", "content": "

Yes, these are the result of a single model and single fold.

", "votes": 0, "replies": [] } ] }, { "id": 1183045, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-02-02T17:58:05.633000", "content": "

The host said that , the test set is also annotated by 3 radiologists but these boxes were unified to one single box by 2 professional radiologists.
\nEither the 3 radiologists or 2 radiologists would be different. Maybe they are different in their annotations.

", "votes": 0, "replies": [ { "id": 1190154, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-02-07T14:15:43.017000", "content": "

Sorry for the late reply. As far as I know, the disagreement will be further judged by the radiologists and finally made a consensus. In my opinion, the difference in annotation distribution between train/test sets makes this competition a more challenging one.

", "votes": 0, "replies": [] } ] }, { "id": 1181146, "author_name": "Awsaf", "author_url": "", "post_date": "2021-02-01T17:20:28.553000", "content": "

I guess we're about to see a huge shake up

", "votes": 0, "replies": [ { "id": 1181643, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-02-02T03:10:00.500000", "content": "

Given the results that we have got now. I feel the same too.

", "votes": 0, "replies": [] } ] }, { "id": 1174174, "author_name": "Manh Lab", "author_url": "", "post_date": "2021-01-28T10:18:08.937000", "content": "

i have the same problem. CV got 0.36 but PL so bad.

", "votes": 0, "replies": [ { "id": 1174200, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-28T10:36:38.760000", "content": "

Maybe we can trust our CV score and pay less attention to the PL.

", "votes": 0, "replies": [] } ] }, { "id": 1170491, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-01-26T08:47:09.157000", "content": "

One of the thing we can understand from this is test set is almost different from the training set . we cannot find a reasonable score on LB as we get in CV

", "votes": 0, "replies": [ { "id": 1170506, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-26T08:56:14.710000", "content": "

Yes, if this is true, it is hard to improve our baseline model, because we do not know which score we could trust, CV or LB. May I ask does your improvement on CV also results in the improvement on LB?

", "votes": 0, "replies": [] }, { "id": 1170591, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-01-26T09:56:36.630000", "content": "

Yes My CV is almost 0.5 for the score of 0.318. We cant trust PublicLB because it is doing only in 300 images

", "votes": 1, "replies": [] }, { "id": 1171040, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-26T15:42:57.837000", "content": "

@morizin, may I ask how you split your whole training set into the cross-validation set?

", "votes": 0, "replies": [] }, { "id": 1173981, "author_name": "arutema47", "author_url": "", "post_date": "2021-01-28T08:02:13.953000", "content": "

I have CV-LB disparencies too.
\nGuess it's a trust-CV competition given the small size of the public set.

", "votes": 2, "replies": [] }, { "id": 1174092, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-01-28T09:05:58.310000", "content": "

@zehuigong I have done StratifiedKFold with labels

", "votes": 0, "replies": [] }, { "id": 1174204, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-28T10:40:12.593000", "content": "

Got it, thanks for your sharing.

", "votes": 0, "replies": [] }, { "id": 1174231, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-01-28T11:00:40.483000", "content": "

but why do you need mine it doesn't correlate with LB

", "votes": 0, "replies": [] }, { "id": 1174268, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-28T11:18:39.610000", "content": "

Now, I think that many of us have encountered this issue, the CV-LB discrepancies.

", "votes": 0, "replies": [] }, { "id": 1181075, "author_name": "adamson", "author_url": "", "post_date": "2021-02-01T16:26:44.103000", "content": "

Are you measuring CV mAP against the raw ground truth? Or against NMS/WBF-transformed boxes?

", "votes": 0, "replies": [] }, { "id": 1181085, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-02-01T16:32:11.173000", "content": "

I checked both with and without never correlates

", "votes": 0, "replies": [] } ] }, { "id": 1174651, "author_name": "", "author_url": "", "post_date": "2021-01-28T16:22:11.443000", "content": "", "votes": 0, "replies": [ { "id": 1175298, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-01-29T04:32:06.910000", "content": "

Have you tried different cross-validation set divisions, i.e., generating different 5 fold cross-validation?

", "votes": 0, "replies": [] }, { "id": 1181525, "author_name": "", "author_url": "", "post_date": "2021-02-02T00:38:42.497000", "content": "", "votes": 0, "replies": [] }, { "id": 1181646, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-02-02T03:12:23.610000", "content": "

Of all the divisions you make, none of them show a correlation between CV and LB score?

", "votes": 0, "replies": [] }, { "id": 1182975, "author_name": "", "author_url": "", "post_date": "2021-02-02T17:04:53.377000", "content": "", "votes": 0, "replies": [] }, { "id": 1182987, "author_name": "ZehuiGong", "author_url": "", "post_date": "2021-02-02T17:11:27.850000", "content": "

May I ask what kind of detection architecture(algorithm) are you experimenting with?

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1170154": "Hi, everyone. I found an issue here. My scores on CV vs. LB can not correspond to each other. Below is an example:\nCV: 0.332 --> LB: 0.217\nCV: 0.350 --> LB: 0.173\nCV: 0.331 -->LB: 0.179\nI am confused about the correlation between CV and LB. How are your scores on CV vs. LB? Posting mine below.", "1192229": "Hi, Are your CV include \"No findings\" class and have you apply \"2 class filtering\" on your LB?", "1188203": "> CV: 0.332 --> LB: 0.217\nCV: 0.350 --> LB: 0.173\nCV: 0.331 -->LB: 0.179\n\nAre these single models? @zehuigong ", "1183045": "The host said that , the test set is also annotated by 3 radiologists but these boxes were unified to one single box by 2 professional radiologists.\nEither the 3 radiologists or 2 radiologists would be different. Maybe they are different in their annotations.", "1181146": "I guess we're about to see a huge shake up", "1174174": "i have the same problem. CV got 0.36 but PL so bad. ", "1170491": "One of the thing we can understand from this is test set is almost different from the training set . we cannot find a reasonable score on LB as we get in CV", "1174651": "" } }