We scored 0.183 on public lb, but our private lb is 0.398, which is very huge drop.
\nAny clues on this? Our past submission shows similar symptoms.
https://www.kaggle.com/bleaf21/submission-ensemble-02tx-bh
", "messages": [ { "id": 1061362, "postDate": "2020-10-27T00:47:49.983Z", "content": "We scored 0.183 on public lb, but our private lb is 0.398, which is very huge drop.
\nAny clues on this? Our past submission shows similar symptoms.
https://www.kaggle.com/bleaf21/submission-ensemble-02tx-bh
", "rawMarkdown": "We scored 0.183 on public lb, but our private lb is 0.398, which is very huge drop. \nAny clues on this? Our past submission shows similar symptoms. \n\n- Checked high correlation with our CV and public lb and it seems like current lb shows high correlation with public lb as well.\n- Used sample submission.csv, merged our result csv based on sample submission.csv ids.\n- Used DEBUG mode, testing only 25 samples during notebook saving session if test.csv length is 146853. Assume this will be not activated on submission phase.\n- Attached our submission notebook below !!!\n\nhttps://www.kaggle.com/bleaf21/submission-ensemble-02tx-bh ", "votes": 9 }, { "id": 1061722, "postDate": "2020-10-27T09:00:34.163Z", "content": "I think it's because you didn't install gdcm. without gdcm, dicom loading error happens and your prediction will stop, then most of your submission will be filled with 0.5. Try to put the 1st line of the baseline notebook.
", "rawMarkdown": " I think it's because you didn't install gdcm. without gdcm, dicom loading error happens and your prediction will stop, then most of your submission will be filled with 0.5. Try to put the 1st line of [the baseline notebook](https://www.kaggle.com/khyeh0719/cnn-gru-baseline-stage2-train-inference).", "votes": 6, "replies": [ { "id": 1061765, "postDate": "2020-10-27T10:02:14.500Z", "content": "Thank you for the reply, I will try this!
", "rawMarkdown": "Thank you for the reply, I will try this!" }, { "id": 1061848, "postDate": "2020-10-27T11:23:15.033Z", "content": "@philculliton , @juliaelliott , dear staffs, we are now trying late submission only adding gdcm install lines. If this was the problem, virtual environment for competition submission notebook should have included appropriate gdcm packages in the first place. We did check there was no problem with public test set and prepared based on the public test set without any clues something was going wrong with our private test set. If there was any messages about the error, we should have known the issue during our previous submissions.
\nStill we need to confirm this, we would like to ask you about what action will be made if really gdcm version issue and installation was the problem. I would like to mention this problem before it is too late.
\nI will upload comment again as soon as the result comes out.
I'm sorry but I think it's too late already. All the winners did a huge effort to make a successful submission before the deadline and you didn't. fillna(0.5) is a fatal mistake of you which made you not find the problem. This problem was discussed here and yuval's strategy is correct.
", "rawMarkdown": "I'm sorry but I think it's too late already. All the winners did a huge effort to make a successful submission before the deadline and you didn't. fillna(0.5) is a fatal mistake of you which made you not find the problem. This problem was discussed [here](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/188006) and yuval's strategy is correct.", "votes": 7 } ] }, { "id": 1061706, "postDate": "2020-10-27T08:34:02.853Z", "content": "Can you make your inputs visible as well? I am trying to diagnose it.
", "rawMarkdown": "Can you make your inputs visible as well? I am trying to diagnose it. ", "votes": 3, "replies": [ { "id": 1061768, "postDate": "2020-10-27T10:04:33.037Z", "content": "Perhaps I am not following. In what type for visibile inputs will be helpful for diagnosing? We are using test dcms in test folder for feature extraction(stage1), stack them and apply sequence model for final outputs which include all exam and image labels(stage2)
", "rawMarkdown": "Perhaps I am not following. In what type for visibile inputs will be helpful for diagnosing? We are using test dcms in test folder for feature extraction(stage1), stack them and apply sequence model for final outputs which include all exam and image labels(stage2)" } ] }, { "id": 1061364, "postDate": "2020-10-27T00:49:05.803Z", "content": "That's bizarre, perhaps it would be a good idea to share your notebook and such? It doesn't seem possible this is overfitting or anything like that.
", "rawMarkdown": "That's bizarre, perhaps it would be a good idea to share your notebook and such? It doesn't seem possible this is overfitting or anything like that.", "votes": 3, "replies": [ { "id": 1061389, "postDate": "2020-10-27T01:24:43.920Z", "content": "Thank you for the advice, we changed our submission notebook status as public, and attached it above.
", "rawMarkdown": "Thank you for the advice, we changed our submission notebook status as public, and attached it above. ", "votes": 3 } ] }, { "id": 1062320, "postDate": "2020-10-27T18:23:23.913Z", "content": "@brekkanegg very sorry about this happening after all your effort!
\nI checked my submissions and seems it was happening exactly the same to me (0.181 Public, 0.396 Private), but this stopped on private after installing gdcm.
I think it's because you didn't install gdcm. without gdcm, dicom loading error happens and your prediction will stop, then most of your submission will be filled with 0.5. Try to put the 1st line of the baseline notebook.
", "votes": 6, "replies": [ { "id": 1061765, "author_name": "brekkanegg", "author_url": "", "post_date": "2020-10-27T10:02:14.500000", "content": "Thank you for the reply, I will try this!
", "votes": 0, "replies": [] }, { "id": 1061848, "author_name": "brekkanegg", "author_url": "", "post_date": "2020-10-27T11:23:15.033000", "content": "@philculliton , @juliaelliott , dear staffs, we are now trying late submission only adding gdcm install lines. If this was the problem, virtual environment for competition submission notebook should have included appropriate gdcm packages in the first place. We did check there was no problem with public test set and prepared based on the public test set without any clues something was going wrong with our private test set. If there was any messages about the error, we should have known the issue during our previous submissions.
\nStill we need to confirm this, we would like to ask you about what action will be made if really gdcm version issue and installation was the problem. I would like to mention this problem before it is too late.
\nI will upload comment again as soon as the result comes out.
I'm sorry but I think it's too late already. All the winners did a huge effort to make a successful submission before the deadline and you didn't. fillna(0.5) is a fatal mistake of you which made you not find the problem. This problem was discussed here and yuval's strategy is correct.
", "votes": 7, "replies": [] } ] }, { "id": 1061706, "author_name": "sin", "author_url": "", "post_date": "2020-10-27T08:34:02.853000", "content": "Can you make your inputs visible as well? I am trying to diagnose it.
", "votes": 3, "replies": [ { "id": 1061768, "author_name": "brekkanegg", "author_url": "", "post_date": "2020-10-27T10:04:33.037000", "content": "Perhaps I am not following. In what type for visibile inputs will be helpful for diagnosing? We are using test dcms in test folder for feature extraction(stage1), stack them and apply sequence model for final outputs which include all exam and image labels(stage2)
", "votes": 0, "replies": [] } ] }, { "id": 1061364, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-27T00:49:05.803000", "content": "That's bizarre, perhaps it would be a good idea to share your notebook and such? It doesn't seem possible this is overfitting or anything like that.
", "votes": 3, "replies": [ { "id": 1061389, "author_name": "brekkanegg", "author_url": "", "post_date": "2020-10-27T01:24:43.920000", "content": "Thank you for the advice, we changed our submission notebook status as public, and attached it above.
", "votes": 3, "replies": [] } ] }, { "id": 1062320, "author_name": "Darragh", "author_url": "", "post_date": "2020-10-27T18:23:23.913000", "content": "@brekkanegg very sorry about this happening after all your effort!
\nI checked my submissions and seems it was happening exactly the same to me (0.181 Public, 0.396 Private), but this stopped on private after installing gdcm.