{ "id": 186989, "title": "Note on Leaderboard and Private/Public test sets", "url": "/competitions/rsna-str-pulmonary-embolism-detection/discussion/186989", "author_name": "quadcore/Richard Epstein", "post_date": "2020-09-26T21:15:10.282000", "votes": 23, "comment_count": 20, "views": 0, "content": "

I wasted a lot of time, and maybe I can save someone else the same effort.

\n

As far as I can tell, the \"Visible\" test data corresponds to the \"Public\" Leaderboard score. And the \"Hidden\" test data corresponds to the \"Private\" Leaderboard score. Not unexpected, but I had not thought it through.

\n

Many probably realized this already.

\n

The percentages discussed by the organizers are not exact, but they are pretty close:

\n

Visible test patients - 650 (30%)
\nHidden test patients - 1517 (70%)

\n

By Size:
\n70 GB - 23%
\n230 GB - 77%

\n

By Leaderboard Description:
\n28%
\n72%

\n

I have been using the Visible test data and the TPU and creating a submission file. Then I used a Committed Notebook to update the hidden sample_submission.csv file with my public submission file. I wasn't doing inference in my committed notebook (due to speed issues).

\n

When I went to combine that with Inference on the hidden data in the Committed Notebook, I was getting no change in my Leaderboard score. I thought I had a programming error. Then I realized the Visible test data exactly corresponded to the Public Leaderboard score.

\n

Testing showed that any changes to the Hidden studies in the submission file did not change the Public Leaderboard score.

\n

It probably took me too long to realize this.

\n

The important conclusion is:
\n During testing/development you can do everything in non-Committed Notebooks and you will still get feedback on the Leaderboard. This technique has been discussed elsewhere.

\n

You will not get any additional feedback if you combine a public inference submission with a committed hidden inference submission. If your committed inference is quietly failing (but not crashing), but your public inference is correct, you will not know that. For instance, you could be using an outdated model in your committed inference without realizing it.

\n

If you take this route, make sure you check every now and then that your code can complete in a committed notebook for your final submissions.

\n

So code carefully. If your model is fast enough, maybe you want to always run the entire public+private test data in a committed notebook.

\n

A long post (because I wasted a lot of time). Maybe obvious to everybody.

\n

-Rich

", "messages": [ { "id": 1028446, "postDate": "2020-09-26T21:15:10.283Z", "content": "

I wasted a lot of time, and maybe I can save someone else the same effort.

\n

As far as I can tell, the \"Visible\" test data corresponds to the \"Public\" Leaderboard score. And the \"Hidden\" test data corresponds to the \"Private\" Leaderboard score. Not unexpected, but I had not thought it through.

\n

Many probably realized this already.

\n

The percentages discussed by the organizers are not exact, but they are pretty close:

\n

Visible test patients - 650 (30%)
\nHidden test patients - 1517 (70%)

\n

By Size:
\n70 GB - 23%
\n230 GB - 77%

\n

By Leaderboard Description:
\n28%
\n72%

\n

I have been using the Visible test data and the TPU and creating a submission file. Then I used a Committed Notebook to update the hidden sample_submission.csv file with my public submission file. I wasn't doing inference in my committed notebook (due to speed issues).

\n

When I went to combine that with Inference on the hidden data in the Committed Notebook, I was getting no change in my Leaderboard score. I thought I had a programming error. Then I realized the Visible test data exactly corresponded to the Public Leaderboard score.

\n

Testing showed that any changes to the Hidden studies in the submission file did not change the Public Leaderboard score.

\n

It probably took me too long to realize this.

\n

The important conclusion is:
\n During testing/development you can do everything in non-Committed Notebooks and you will still get feedback on the Leaderboard. This technique has been discussed elsewhere.

\n

You will not get any additional feedback if you combine a public inference submission with a committed hidden inference submission. If your committed inference is quietly failing (but not crashing), but your public inference is correct, you will not know that. For instance, you could be using an outdated model in your committed inference without realizing it.

\n

If you take this route, make sure you check every now and then that your code can complete in a committed notebook for your final submissions.

\n

So code carefully. If your model is fast enough, maybe you want to always run the entire public+private test data in a committed notebook.

\n

A long post (because I wasted a lot of time). Maybe obvious to everybody.

\n

-Rich

", "rawMarkdown": "I wasted a lot of time, and maybe I can save someone else the same effort.\n\nAs far as I can tell, the \"Visible\" test data corresponds to the \"Public\" Leaderboard score. And the \"Hidden\" test data corresponds to the \"Private\" Leaderboard score. Not unexpected, but I had not thought it through.\n\nMany probably realized this already.\n\nThe percentages discussed by the organizers are not exact, but they are pretty close:\n\nVisible test patients - 650 (30%)\nHidden test patients - 1517 (70%)\n\nBy Size:\n70 GB - 23%\n230 GB - 77%\n\nBy Leaderboard Description:\n28%\n72%\n\nI have been using the Visible test data and the TPU and creating a submission file. Then I used a Committed Notebook to update the hidden sample_submission.csv file with my public submission file. I wasn't doing inference in my committed notebook (due to speed issues).\n\nWhen I went to combine that with Inference on the hidden data in the Committed Notebook, I was getting no change in my Leaderboard score. I thought I had a programming error. Then I realized the Visible test data exactly corresponded to the Public Leaderboard score.\n\nTesting showed that any changes to the Hidden studies in the submission file did not change the Public Leaderboard score.\n\nIt probably took me too long to realize this.\n\nThe important conclusion is:\n During testing/development you can do everything in non-Committed Notebooks and you will still get feedback on the Leaderboard. This technique has been discussed elsewhere.\n\n You will not get any additional feedback if you combine a public inference submission with a committed hidden inference submission. If your committed inference is quietly failing (but not crashing), but your public inference is correct, you will not know that. For instance, you could be using an outdated model in your committed inference without realizing it.\n\nIf you take this route, make sure you check every now and then that your code can complete in a committed notebook for your final submissions.\n\nSo code carefully. If your model is fast enough, maybe you want to always run the entire public+private test data in a committed notebook.\n\nA long post (because I wasted a lot of time). Maybe obvious to everybody.\n\n-Rich", "votes": 22 }, { "id": 1028975, "postDate": "2020-09-27T11:16:28.720Z", "content": "

Not obvious and very helpful post, thank you!

", "rawMarkdown": "Not obvious and very helpful post, thank you!", "votes": 1 }, { "id": 1044577, "postDate": "2020-10-09T23:32:33.043Z", "content": "

So does that mean 650 studies out of 1517 studies in the private dataset are actually the same as the public dataset. If that is so can't we save time by having a submission.csv file of 650 public studies created locally and inferring on (1517-650=867) private studies and combining both csvs?

", "rawMarkdown": "So does that mean 650 studies out of 1517 studies in the private dataset are actually the same as the public dataset. If that is so can't we save time by having a submission.csv file of 650 public studies created locally and inferring on (1517-650=867) private studies and combining both csvs?", "votes": 2, "replies": [ { "id": 1044636, "postDate": "2020-10-10T02:05:33.243Z", "content": "

I believe the full test set is 650+1517=2167 studies. But the rest of your idea should work. Combine public 650 with a committed inference on the other 1517.

", "rawMarkdown": "I believe the full test set is 650+1517=2167 studies. But the rest of your idea should work. Combine public 650 with a committed inference on the other 1517.", "votes": 2 }, { "id": 1044642, "postDate": "2020-10-10T02:16:20.547Z", "content": "

Oh, it is 2167. That makes more sense with my rerun time. Thanks a lot :)

", "rawMarkdown": "Oh, it is 2167. That makes more sense with my rerun time. Thanks a lot :)", "votes": 1 }, { "id": 1056600, "postDate": "2020-10-21T21:58:39.397Z", "content": "

Hmm, this must be based on experience. I couldn't see anywhere it says the re-run will be with the private and public test sets combined.

", "rawMarkdown": "Hmm, this must be based on experience. I couldn't see anywhere it says the re-run will be with the private and public test sets combined.", "replies": [ { "id": 1057208, "postDate": "2020-10-22T13:24:39.560Z", "content": "

This isn't quite explicity stated, but I think it must be true.

\n

If you make a submission using only the public test data (and averages for the private test data to make a valid file), you get a public leaderboard score.

\n

Obviously, to get a private leaderboard score, you'll need to predict the private test set.

\n

So the final \"commit\" notebook must be submitting both private and public.

\n

I do not know what would happen if your final \"commit\" notebook only inferenced the private test set and filled in blanks/averages for the public test set. Is your private leaderboard score only based on the private test set? I think so, but I don't know if that is guaranteed.

\n

So I would make sure your \"commit\" notebook that you pick for the final leaderboard includes predictions for the entire hidden sample_submission.csv file.

\n

Also, since you might have a good public leaderboard score on a submission that does not include inference on the private test set, make sure you pick which submissions to score for the private leaderboard carefully.

", "rawMarkdown": "This isn't quite explicity stated, but I think it must be true.\n\nIf you make a submission using only the public test data (and averages for the private test data to make a valid file), you get a public leaderboard score.\n\nObviously, to get a private leaderboard score, you'll need to predict the private test set.\n\nSo the final \"commit\" notebook must be submitting both private and public.\n\nI do not know what would happen if your final \"commit\" notebook only inferenced the private test set and filled in blanks/averages for the public test set. Is your private leaderboard score only based on the private test set? I think so, but I don't know if that is guaranteed.\n\nSo I would make sure your \"commit\" notebook that you pick for the final leaderboard includes predictions for the entire hidden sample_submission.csv file.\n\nAlso, since you might have a good public leaderboard score on a submission that does not include inference on the private test set, make sure you pick which submissions to score for the private leaderboard carefully." } ] } ] }, { "id": 1031492, "postDate": "2020-09-29T13:41:58.943Z", "content": "

Nice post..

", "rawMarkdown": "Nice post..", "votes": -1 }, { "id": 1031138, "postDate": "2020-09-29T09:06:43.377Z", "content": "

good job ! good

", "rawMarkdown": "good job ! good", "votes": -7 }, { "id": 1056679, "postDate": "2020-10-22T01:15:03.727Z", "content": "

Does the public test dataset differ than what is used for our LB scores before the competition end?

\n

Maybe this is obvious, but I noticed something odd. I cached the sample_submission.csv from the public dataset. Then, I submitted that csv and got a Submission Scoring Error. Note, this is different than referencing '../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv' and submitting that, which works. In other words, sample_submission.csv is different in the public test dataset than the leaderboard run.

", "rawMarkdown": "Does the public test dataset differ than what is used for our LB scores before the competition end?\n\nMaybe this is obvious, but I noticed something odd. I cached the sample_submission.csv from the public dataset. Then, I submitted that csv and got a Submission Scoring Error. Note, this is different than referencing '../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv' and submitting that, which works. In other words, sample_submission.csv is different in the public test dataset than the leaderboard run.", "replies": [ { "id": 1057158, "postDate": "2020-10-22T12:30:20.950Z", "content": "

Your public leaderboard score is based on the public test set, but…

\n

To get a score, you must send in a full submission - based on the private sample submission and including entries for both the public and private submission.

\n

So by sending in just the public submission, you are not sending in a valid file.

\n

If you take the full hidden sample submission and update the public part with your labels from your public submission, and submit that (in your committed notebook), you will get a public score.

\n

Of course, there are two factors to remember:

\n
    \n
  1. If you don't inference the private test set, your private LB score will be terrible.
    2. \n
  2. Top contenders must also meet the consistency restraints - which are not reflected in the public leaderboard.
    3. \n
\n

-Rich

", "rawMarkdown": "Your public leaderboard score is based on the public test set, but...\n\nTo get a score, you must send in a full submission - based on the private sample submission and including entries for both the public and private submission.\n\nSo by sending in just the public submission, you are not sending in a valid file.\n\nIf you take the full hidden sample submission and update the public part with your labels from your public submission, and submit that (in your committed notebook), you will get a public score.\n\nOf course, there are two factors to remember:\n 1. If you don't inference the private test set, your private LB score will be terrible.\n 2. Top contenders must also meet the consistency restraints - which are not reflected in the public leaderboard.\n\n-Rich\n", "votes": 1 } ] }, { "id": 1044961, "postDate": "2020-10-10T08:51:31.167Z", "content": "

Nice post

", "rawMarkdown": "Nice post\n" }, { "id": 1030940, "postDate": "2020-09-29T05:33:25.623Z", "content": "

Thanks for this insight!

\n
\n

you will still get feedback on the Leaderboard

\n
\n

How so? This may be a dumb question, but don't you have to Save & Run All + Submit Predictions to get the LB score? I don't see the ground truths on the test set anywhere, so I'm not sure how you can get feedback without going through the full submission.

", "rawMarkdown": "Thanks for this insight!\n\n> you will still get feedback on the Leaderboard\n\nHow so? This may be a dumb question, but don't you have to Save & Run All + Submit Predictions to get the LB score? I don't see the ground truths on the test set anywhere, so I'm not sure how you can get feedback without going through the full submission.", "replies": [ { "id": 1031272, "postDate": "2020-09-29T11:02:47.630Z", "content": "

The public leader board reports your score on the public test data. So you only have to inference the public test data to see how you are doing.

\n

You can inference the public data and save a submission file. Then use another notebook that updates the hidden sample submission with your inference file. That notebook needs to be committed.

", "rawMarkdown": "The public leader board reports your score on the public test data. So you only have to inference the public test data to see how you are doing. \n\nYou can inference the public data and save a submission file. Then use another notebook that updates the hidden sample submission with your inference file. That notebook needs to be committed. " }, { "id": 1031870, "postDate": "2020-09-29T18:36:03.057Z", "content": "

Gotcha, thanks!

", "rawMarkdown": "Gotcha, thanks!", "votes": -1 }, { "id": 1037625, "postDate": "2020-10-05T07:15:19.423Z", "content": "

I still have some doubts. So I did inference on my public test dataset and got a submission.csv. How can we use that submission file to update the hidden sample submission file? Won't the hidden submission file be having different ids compared to our inference submission file?

", "rawMarkdown": "I still have some doubts. So I did inference on my public test dataset and got a submission.csv. How can we use that submission file to update the hidden sample submission file? Won't the hidden submission file be having different ids compared to our inference submission file?", "replies": [ { "id": 1037978, "postDate": "2020-10-05T13:05:59.483Z", "content": "

The hidden submission has the public ids and the private ids combined. You need to submit something with both, but your public Leaderboard score only shows you the result of the public ids.

\n

So the private submission file is \"half\" filled in with your public inference and \"half\" left as defaults just to keep the format correct.

\n

This is great for faster testing. In the end, you must of course send in inference for the private ids also.

", "rawMarkdown": "The hidden submission has the public ids and the private ids combined. You need to submit something with both, but your public Leaderboard score only shows you the result of the public ids.\n\nSo the private submission file is \"half\" filled in with your public inference and \"half\" left as defaults just to keep the format correct.\n\nThis is great for faster testing. In the end, you must of course send in inference for the private ids also.", "votes": 1 }, { "id": 1038011, "postDate": "2020-10-05T13:33:34.577Z", "content": "

Thanks.
\nSome doubts. I have a prediction file of the public test as \"submission.csv\". So is this how you do it?

\n
inference_df = pd.read_csv('../input/submission-rsna-pulmonary/submission.csv')\nsub=pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv')\nfor i in tqdm(range(len(inference_df))):\n    inference_id = inference_df.iloc[i]['id']\n    sub.loc[sub.id==inference_id,'label']=inference_df.iloc[i]['label']\nsub.to_csv('submission.csv', index = False)\n
\n

This will take about 1.5 hour. Is there a better way?

", "rawMarkdown": "Thanks. \nSome doubts. I have a prediction file of the public test as \"submission.csv\". So is this how you do it?\n```\ninference_df = pd.read_csv('../input/submission-rsna-pulmonary/submission.csv')\nsub=pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv')\nfor i in tqdm(range(len(inference_df))):\n inference_id = inference_df.iloc[i]['id']\n sub.loc[sub.id==inference_id,'label']=inference_df.iloc[i]['label']\nsub.to_csv('submission.csv', index = False)\n```\nThis will take about 1.5 hour. Is there a better way? \n" }, { "id": 1038081, "postDate": "2020-10-05T14:32:32.067Z", "content": "

Here's how I do it (not sure where I first saw the idea). Basically, do a merge and copy the old label to the new column if the new column is NAN. Then delete the old label and rename the new label correctly:

\n

submission = pd.read_csv(mypath+'/sample_submission.csv')
\nsubmission_prebuilt = pd.read_csv(submission_prebuilt_file)

\n

submission = pd.merge(submission,submission_prebuilt,on='id',how='left')

\n

submission['label_y'] = submission['label_y'].fillna(submission['label_x'])
\nsubmission.drop(['label_x'],inplace=True,axis=1)
\nsubmission = submission.rename(columns={'label_y':'label'})

", "rawMarkdown": "Here's how I do it (not sure where I first saw the idea). Basically, do a merge and copy the old label to the new column if the new column is NAN. Then delete the old label and rename the new label correctly:\n\nsubmission = pd.read_csv(mypath+'/sample_submission.csv')\nsubmission_prebuilt = pd.read_csv(submission_prebuilt_file)\n\nsubmission = pd.merge(submission,submission_prebuilt,on='id',how='left')\n \nsubmission['label_y'] = submission['label_y'].fillna(submission['label_x'])\nsubmission.drop(['label_x'],inplace=True,axis=1)\nsubmission = submission.rename(columns={'label_y':'label'})\n" }, { "id": 1038093, "postDate": "2020-10-05T14:41:50.240Z", "content": "

Thanks a lot for your help. I just tried using update and it worked.

\n
sub=pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv')\nsub.set_index('id', inplace=True)\n\ninference_df = pd.read_csv('../input/submission-rsna-pulmonary/submission.csv')\ninference_df.set_index('id', inplace=True)\n\nsub.update(inference_df)\nsub.reset_index(inplace=True)\n
", "rawMarkdown": "Thanks a lot for your help. I just tried using update and it worked.\n```\nsub=pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv')\nsub.set_index('id', inplace=True)\n\ninference_df = pd.read_csv('../input/submission-rsna-pulmonary/submission.csv')\ninference_df.set_index('id', inplace=True)\n\nsub.update(inference_df)\nsub.reset_index(inplace=True)\n```\n" } ] } ] }, { "id": 1030201, "postDate": "2020-09-28T13:24:51.967Z", "content": "

Just tried it, you are right.
\nNice finding! Thanks!

", "rawMarkdown": "Just tried it, you are right. \nNice finding! Thanks!" } ], "comments": [ { "id": 1028975, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-09-27T11:16:28.720000", "content": "

Not obvious and very helpful post, thank you!

", "votes": 1, "replies": [] }, { "id": 1044577, "author_name": "Jose", "author_url": "", "post_date": "2020-10-09T23:32:33.043000", "content": "

So does that mean 650 studies out of 1517 studies in the private dataset are actually the same as the public dataset. If that is so can't we save time by having a submission.csv file of 650 public studies created locally and inferring on (1517-650=867) private studies and combining both csvs?

", "votes": 2, "replies": [ { "id": 1044636, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-10-10T02:05:33.243000", "content": "

I believe the full test set is 650+1517=2167 studies. But the rest of your idea should work. Combine public 650 with a committed inference on the other 1517.

", "votes": 2, "replies": [] }, { "id": 1044642, "author_name": "Jose", "author_url": "", "post_date": "2020-10-10T02:16:20.547000", "content": "

Oh, it is 2167. That makes more sense with my rerun time. Thanks a lot :)

", "votes": 1, "replies": [] }, { "id": 1056600, "author_name": "Philip Dhingra", "author_url": "", "post_date": "2020-10-21T21:58:39.397000", "content": "

Hmm, this must be based on experience. I couldn't see anywhere it says the re-run will be with the private and public test sets combined.

", "votes": 0, "replies": [ { "id": 1057208, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-10-22T13:24:39.560000", "content": "

This isn't quite explicity stated, but I think it must be true.

\n

If you make a submission using only the public test data (and averages for the private test data to make a valid file), you get a public leaderboard score.

\n

Obviously, to get a private leaderboard score, you'll need to predict the private test set.

\n

So the final \"commit\" notebook must be submitting both private and public.

\n

I do not know what would happen if your final \"commit\" notebook only inferenced the private test set and filled in blanks/averages for the public test set. Is your private leaderboard score only based on the private test set? I think so, but I don't know if that is guaranteed.

\n

So I would make sure your \"commit\" notebook that you pick for the final leaderboard includes predictions for the entire hidden sample_submission.csv file.

\n

Also, since you might have a good public leaderboard score on a submission that does not include inference on the private test set, make sure you pick which submissions to score for the private leaderboard carefully.

", "votes": 0, "replies": [] } ] } ] }, { "id": 1031492, "author_name": "Amol Ambkar", "author_url": "", "post_date": "2020-09-29T13:41:58.943000", "content": "

Nice post..

", "votes": -1, "replies": [] }, { "id": 1031138, "author_name": "Naim Mhedhbi", "author_url": "", "post_date": "2020-09-29T09:06:43.377000", "content": "

good job ! good

", "votes": -7, "replies": [] }, { "id": 1056679, "author_name": "Philip Dhingra", "author_url": "", "post_date": "2020-10-22T01:15:03.727000", "content": "

Does the public test dataset differ than what is used for our LB scores before the competition end?

\n

Maybe this is obvious, but I noticed something odd. I cached the sample_submission.csv from the public dataset. Then, I submitted that csv and got a Submission Scoring Error. Note, this is different than referencing '../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv' and submitting that, which works. In other words, sample_submission.csv is different in the public test dataset than the leaderboard run.

", "votes": 0, "replies": [ { "id": 1057158, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-10-22T12:30:20.950000", "content": "

Your public leaderboard score is based on the public test set, but…

\n

To get a score, you must send in a full submission - based on the private sample submission and including entries for both the public and private submission.

\n

So by sending in just the public submission, you are not sending in a valid file.

\n

If you take the full hidden sample submission and update the public part with your labels from your public submission, and submit that (in your committed notebook), you will get a public score.

\n

Of course, there are two factors to remember:

\n
    \n
  1. If you don't inference the private test set, your private LB score will be terrible.
    2. \n
  2. Top contenders must also meet the consistency restraints - which are not reflected in the public leaderboard.
    3. \n
\n

-Rich

", "votes": 1, "replies": [] } ] }, { "id": 1044961, "author_name": "Raghib Shams", "author_url": "", "post_date": "2020-10-10T08:51:31.167000", "content": "

Nice post

", "votes": 0, "replies": [] }, { "id": 1030940, "author_name": "Philip Dhingra", "author_url": "", "post_date": "2020-09-29T05:33:25.623000", "content": "

Thanks for this insight!

\n
\n

you will still get feedback on the Leaderboard

\n
\n

How so? This may be a dumb question, but don't you have to Save & Run All + Submit Predictions to get the LB score? I don't see the ground truths on the test set anywhere, so I'm not sure how you can get feedback without going through the full submission.

", "votes": 0, "replies": [ { "id": 1031272, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-09-29T11:02:47.630000", "content": "

The public leader board reports your score on the public test data. So you only have to inference the public test data to see how you are doing.

\n

You can inference the public data and save a submission file. Then use another notebook that updates the hidden sample submission with your inference file. That notebook needs to be committed.

", "votes": 0, "replies": [] }, { "id": 1031870, "author_name": "Philip Dhingra", "author_url": "", "post_date": "2020-09-29T18:36:03.057000", "content": "

Gotcha, thanks!

", "votes": -1, "replies": [] }, { "id": 1037625, "author_name": "Jose", "author_url": "", "post_date": "2020-10-05T07:15:19.423000", "content": "

I still have some doubts. So I did inference on my public test dataset and got a submission.csv. How can we use that submission file to update the hidden sample submission file? Won't the hidden submission file be having different ids compared to our inference submission file?

", "votes": 0, "replies": [ { "id": 1037978, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-10-05T13:05:59.483000", "content": "

The hidden submission has the public ids and the private ids combined. You need to submit something with both, but your public Leaderboard score only shows you the result of the public ids.

\n

So the private submission file is \"half\" filled in with your public inference and \"half\" left as defaults just to keep the format correct.

\n

This is great for faster testing. In the end, you must of course send in inference for the private ids also.

", "votes": 1, "replies": [] }, { "id": 1038011, "author_name": "Jose", "author_url": "", "post_date": "2020-10-05T13:33:34.577000", "content": "

Thanks.
\nSome doubts. I have a prediction file of the public test as \"submission.csv\". So is this how you do it?

\n
inference_df = pd.read_csv('../input/submission-rsna-pulmonary/submission.csv')\nsub=pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv')\nfor i in tqdm(range(len(inference_df))):\n    inference_id = inference_df.iloc[i]['id']\n    sub.loc[sub.id==inference_id,'label']=inference_df.iloc[i]['label']\nsub.to_csv('submission.csv', index = False)\n
\n

This will take about 1.5 hour. Is there a better way?

", "votes": 0, "replies": [] }, { "id": 1038081, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-10-05T14:32:32.067000", "content": "

Here's how I do it (not sure where I first saw the idea). Basically, do a merge and copy the old label to the new column if the new column is NAN. Then delete the old label and rename the new label correctly:

\n

submission = pd.read_csv(mypath+'/sample_submission.csv')
\nsubmission_prebuilt = pd.read_csv(submission_prebuilt_file)

\n

submission = pd.merge(submission,submission_prebuilt,on='id',how='left')

\n

submission['label_y'] = submission['label_y'].fillna(submission['label_x'])
\nsubmission.drop(['label_x'],inplace=True,axis=1)
\nsubmission = submission.rename(columns={'label_y':'label'})

", "votes": 0, "replies": [] }, { "id": 1038093, "author_name": "Jose", "author_url": "", "post_date": "2020-10-05T14:41:50.240000", "content": "

Thanks a lot for your help. I just tried using update and it worked.

\n
sub=pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv')\nsub.set_index('id', inplace=True)\n\ninference_df = pd.read_csv('../input/submission-rsna-pulmonary/submission.csv')\ninference_df.set_index('id', inplace=True)\n\nsub.update(inference_df)\nsub.reset_index(inplace=True)\n
", "votes": 0, "replies": [] } ] } ] }, { "id": 1030201, "author_name": "Tsai29", "author_url": "", "post_date": "2020-09-28T13:24:51.967000", "content": "

Just tried it, you are right.
\nNice finding! Thanks!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1028446": "I wasted a lot of time, and maybe I can save someone else the same effort.\n\nAs far as I can tell, the \"Visible\" test data corresponds to the \"Public\" Leaderboard score. And the \"Hidden\" test data corresponds to the \"Private\" Leaderboard score. Not unexpected, but I had not thought it through.\n\nMany probably realized this already.\n\nThe percentages discussed by the organizers are not exact, but they are pretty close:\n\nVisible test patients - 650 (30%)\nHidden test patients - 1517 (70%)\n\nBy Size:\n70 GB - 23%\n230 GB - 77%\n\nBy Leaderboard Description:\n28%\n72%\n\nI have been using the Visible test data and the TPU and creating a submission file. Then I used a Committed Notebook to update the hidden sample_submission.csv file with my public submission file. I wasn't doing inference in my committed notebook (due to speed issues).\n\nWhen I went to combine that with Inference on the hidden data in the Committed Notebook, I was getting no change in my Leaderboard score. I thought I had a programming error. Then I realized the Visible test data exactly corresponded to the Public Leaderboard score.\n\nTesting showed that any changes to the Hidden studies in the submission file did not change the Public Leaderboard score.\n\nIt probably took me too long to realize this.\n\nThe important conclusion is:\n During testing/development you can do everything in non-Committed Notebooks and you will still get feedback on the Leaderboard. This technique has been discussed elsewhere.\n\n You will not get any additional feedback if you combine a public inference submission with a committed hidden inference submission. If your committed inference is quietly failing (but not crashing), but your public inference is correct, you will not know that. For instance, you could be using an outdated model in your committed inference without realizing it.\n\nIf you take this route, make sure you check every now and then that your code can complete in a committed notebook for your final submissions.\n\nSo code carefully. If your model is fast enough, maybe you want to always run the entire public+private test data in a committed notebook.\n\nA long post (because I wasted a lot of time). Maybe obvious to everybody.\n\n-Rich", "1028975": "Not obvious and very helpful post, thank you!", "1044577": "So does that mean 650 studies out of 1517 studies in the private dataset are actually the same as the public dataset. If that is so can't we save time by having a submission.csv file of 650 public studies created locally and inferring on (1517-650=867) private studies and combining both csvs?", "1031492": "Nice post..", "1031138": "good job ! good", "1056679": "Does the public test dataset differ than what is used for our LB scores before the competition end?\n\nMaybe this is obvious, but I noticed something odd. I cached the sample_submission.csv from the public dataset. Then, I submitted that csv and got a Submission Scoring Error. Note, this is different than referencing '../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv' and submitting that, which works. In other words, sample_submission.csv is different in the public test dataset than the leaderboard run.", "1044961": "Nice post\n", "1030940": "Thanks for this insight!\n\n> you will still get feedback on the Leaderboard\n\nHow so? This may be a dumb question, but don't you have to Save & Run All + Submit Predictions to get the LB score? I don't see the ground truths on the test set anywhere, so I'm not sure how you can get feedback without going through the full submission.", "1030201": "Just tried it, you are right. \nNice finding! Thanks!" } }