In this notebook, I tested some basic assumptions about test dataset.
\nI demonstrated that following assumptions are all TRUE.
[updated at 2022-12-04]
\n[updated at 2022-12-05]
\n[updated at 2022-12-15]
\nI am happy if anyone correct me if I am wrong.
\nI am also very happy if anyone share us other assumptions/hypothesis about test dataset.
In this notebook, I tested some basic assumptions about test dataset.
\nI demonstrated that following assumptions are all TRUE.
[updated at 2022-12-04]
\n[updated at 2022-12-05]
\n[updated at 2022-12-15]
\nI am happy if anyone correct me if I am wrong.
\nI am also very happy if anyone share us other assumptions/hypothesis about test dataset.
\n\nAll patients have CC and MLO images for both sides
\n
This is surprising, as there seemed to be examinations in the train set where at least one of the views were missing (e.g., 5 DICOM files, 3 L-CC, 1 L-MLO and 1 R-MLO, but no R-CC).
Thank you. Is this true: 'There are new machine IDs in test dataset' ?
", "rawMarkdown": "Thank you. Is this true: '*There are new machine IDs in test dataset'* ?", "votes": 1, "replies": [ { "id": 2102062, "postDate": "2023-01-16T11:09:13.710Z", "content": "Yes, it is.
\nPlease check cell 10 of my notebook.
\nIt was surprising for me.
thank you.
", "rawMarkdown": "thank you.", "votes": 1 } ] } ] }, { "id": 2055387, "postDate": "2022-12-05T03:52:18.257Z", "content": "all zero submission, LB = ???
\nall one submission, LB = ???
\nfrom this we can compute number/ratio of pos images (for public)
All zero submission? That's just zero, right ?
", "rawMarkdown": "All zero submission? That's just zero, right ?", "votes": 1 }, { "id": 2055406, "postDate": "2022-12-05T04:10:56.487Z", "content": "maybe more correctly:
\nnow we \"know\" the label of the 4 test images (from your models or additional probing), then you can work out number/ratio of pos images (for public)
", "rawMarkdown": "maybe more correctly:\n1. we are given 4 test images (verify that these are in public)\n2. submit the score of 4 test images = 0.5, rest of test images with score = 0.\n3. submit the score of 4 test images = 0.5, rest of test images with score = 1.\n\nnow we \"know\" the label of the 4 test images (from your models or additional probing), then you can work out number/ratio of pos images (for public)\n", "votes": 4 }, { "id": 2055409, "postDate": "2022-12-05T04:15:59.743Z", "content": "You'd have to do some shenanigans with your personal leaderboard I think to zero in on an accurate number with only two decimals to work with. Or maybe there's some way to do this in parts. Haven't really thought about it.
\nI think it's worth doing though, will look at it tomorrow. The incidence should be roughly around .02 or the split was messed with.
", "rawMarkdown": "You'd have to do some shenanigans with your personal leaderboard I think to zero in on an accurate number with only two decimals to work with. Or maybe there's some way to do this in parts. Haven't really thought about it.\n\nI think it's worth doing though, will look at it tomorrow. The incidence should be roughly around .02 or the split was messed with.", "votes": 1 }, { "id": 2055567, "postDate": "2022-12-05T07:09:54.360Z", "content": "submitting all ones is enough btw
\nfollow the formula
\nmean_cancer = 0.5*LB_score / (1- 0.5*LB_score)
which follows from
\nF1 = 2TP / (2TP + FN + FP)
when using that FN = 0 and FP = 1-TP in a submission of all ones
", "rawMarkdown": "submitting all ones is enough btw\n\nfollow the formula \n\n`mean_cancer = 0.5*LB_score / (1- 0.5*LB_score)` \n\nwhich follows from \n\n`F1 = 2TP / (2TP + FN + FP)`\n\nwhen using that FN = 0 and FP = 1-TP in a submission of all ones", "votes": 4 }, { "id": 2055591, "postDate": "2022-12-05T07:56:22.283Z", "content": "Share my result
\nall one LB: 0.04
\nmean_cancer: ~0.02
Thank you for testing!
\nConsidering the decimal rounding, LB score is 0.035-0.045, and positive rate would be 0.0178-0.0230.
\nThe value is similar to that of train set (0.0206).
i am thinking of something like this:
\nfor each machine id sort the prediction by score.
\nthreshold by score value or score ranking?
a note on threshold by score. binary threshold may not be the best. there are other way to re calibrate the prediction.
", "rawMarkdown": "i am thinking of something like this:\n\nfor each machine id sort the prediction by score. \nthreshold by score value or score ranking?\n\n---\n\na note on threshold by score. binary threshold may not be the best. there are other way to re calibrate the prediction.", "votes": 1 }, { "id": 2055836, "postDate": "2022-12-05T13:35:30.793Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2060722, "postDate": "2022-12-10T10:50:22.143Z", "content": "@tomooinubushi LB decimal rounding is 0.4 = [0.0400000; 0.04999999] though
", "rawMarkdown": "@tomooinubushi LB decimal rounding is 0.4 = [0.0400000; 0.04999999] though", "votes": 1 }, { "id": 2060840, "postDate": "2022-12-10T13:54:31.177Z", "content": "Thank you for pointing it out! I didn't know that.
\nSo, the positive rate is around 0.0204-0.0256, which is still similar to that of train set.
test_machine_49_count = len(test_df.query(\"machine_id == 49\"))
\ntest_len = len(test_df)
\ntest_machine_49_count/test_len > 0.40 is True. (.43 in traindf)
Thank you for your comment. It was True.
\nSo, we don't need to be afraid too much of hidden new machine IDs.
you can verify this:
\nall machine_id has at least some cancer images at hidden test
Do you mean all machines have at least one cancer image?
\nHow can I test it? Is it based on LB score values?
Yeah, I'm kinda curious how he figured this out as well. Assuming the distributions are roughly the same, you probably only have to check 190 and 197. It's possible that machine_id is simply not a signal at all and we've just got a weird split with regards to those. Not sure why'd they include it in the test data if it wasn't a signal, tbh. But I'm also confused as to why density isn't in the test set.
", "rawMarkdown": "Yeah, I'm kinda curious how he figured this out as well. Assuming the distributions are roughly the same, you probably only have to check 190 and 197. It's possible that machine_id is simply not a signal at all and we've just got a weird split with regards to those. Not sure why'd they include it in the test data if it wasn't a signal, tbh. But I'm also confused as to why density isn't in the test set.\n\n" }, { "id": 2055790, "postDate": "2022-12-05T12:54:25.550Z", "content": "I thought he had some way of doing it without submitting for each machine id, but it occurs to me that there is some value in actually submitting for each one. It would be interesting to see if the scors line up with the traindf.
\nEg, using train
\n k train[].unique():\n (k, pfbeta(train[], [ x != k x train[]]))\n \n \n \n \n \n \n \n \n \n \nBe good to verify roughly the count of how many new machine_ids we see as well and there over all score.,
", "rawMarkdown": "I thought he had some way of doing it without submitting for each machine id, but it occurs to me that there is some value in actually submitting for each one. It would be interesting to see if the scors line up with the traindf.\n\nEg, using train\n\n```python\nfor k in train['machine_id'].unique():\n print(k, pfbeta(train['cancer'], [0 if x != k else 1 for x in train['machine_id']]))\n```\n\n```python\n29 0.03416445623342175\n21 0.03283932188932722\n216 0.005218525766470972\n93 0.009111617312072893\n49 0.04974277960059951\n48 0.03631936694734706\n170 0.02210475732820759\n210 0\n190 0.007674597083653109\n197 0\n```\n\nBe good to verify roughly the count of how many new machine_ids we see as well and there over all score.," }, { "id": 2055814, "postDate": "2022-12-05T13:10:07.027Z", "content": "for 210 0, 197 0, i suspect the (few) pos cases are in the test
", "rawMarkdown": "for 210 0, 197 0, i suspect the (few) pos cases are in the test" }, { "id": 2056309, "postDate": "2022-12-06T01:41:50.247Z", "content": "Scores don't seem to be correlating for machine_ids, so either incidence is off or distribution of machine usage.
", "rawMarkdown": "Scores don't seem to be correlating for machine_ids, so either incidence is off or distribution of machine usage." } ] }, { "id": 2054476, "postDate": "2022-12-04T07:16:59.677Z", "content": "As with train dataset, the average site id was <1.5. I wish they were all 2;)
", "rawMarkdown": "As with train dataset, the average site id was <1.5. I wish they were all 2;)", "votes": 1, "replies": [ { "id": 2054524, "postDate": "2022-12-04T08:01:24.697Z", "content": "Thank you for your comment. I tested it, and it was true.
\nDo you think images in site id 2 are easy to diagnose?
Because the site id's for the given test were all 2.
\nI thought I could further increase the score by devising a validation set if they are all 2.
Some other distribution things to check, just to verify it was a random split and not something else:
\nIs mean age around 57.5 at site 1, 60 at site2? Maybe verify the std and nancount as well
\nAre there about 0.000644 AT views? (AT views mean a lesion was found, 10% incidence of cancer in train set)
\nImplants around 5% (often results in lower incidence of cancer due to traits of women who get implants)
I verified mean age and implant rate. Together with the positive rate from LB score, the split seems to be random.
", "rawMarkdown": "I verified mean age and implant rate. Together with the positive rate from LB score, the split seems to be random." }, { "id": 2055763, "postDate": "2022-12-05T12:38:42.587Z", "content": "Yeah, looks good!
\nI'm going to check out the implants being reported only in site 1 as well. I like how you're verifying all the natural assumptions one would make.
\nSome deeper probing on the machine ids is probably a good idea as that is a surprise.
", "rawMarkdown": "Yeah, looks good!\n\nI'm going to check out the implants being reported only in site 1 as well. I like how you're verifying all the natural assumptions one would make.\n\nSome deeper probing on the machine ids is probably a good idea as that is a surprise." } ] }, { "id": 2054654, "postDate": "2022-12-04T10:20:59.223Z", "content": "Thanks @tomooinubushi
\nThat's what I was wondering. Now we know better how to build our CV.
", "rawMarkdown": "Thanks @tomooinubushi \n\n- Patient IDs in train and test sets do not overlap\n- Image IDs in train and test sets do not overlap\n\nThat's what I was wondering. Now we know better how to build our CV.\n", "votes": 2 }, { "id": 2149352, "postDate": "2023-02-18T08:00:03.670Z", "content": "Apologies If I am asking a duplicate question…what is the total number of patients in the hidden test set?
", "rawMarkdown": "Apologies If I am asking a duplicate question...what is the total number of patients in the hidden test set?", "replies": [ { "id": 2149399, "postDate": "2023-02-18T09:07:44.997Z", "content": "\"You can expect roughly 8,000 patients in the hidden test set. \" - from Data section of this competition.
", "rawMarkdown": "\"You can expect roughly 8,000 patients in the hidden test set. \" - from Data section of this competition.", "votes": 2, "replies": [ { "id": 2149439, "postDate": "2023-02-18T09:51:30.327Z", "content": "thanks a lot for clarifying…much appreciated
", "rawMarkdown": "thanks a lot for clarifying...much appreciated " } ] } ] } ], "comments": [ { "id": 2148868, "author_name": "Antti Isosalo", "author_url": "", "post_date": "2023-02-17T18:14:19.580000", "content": "\n\nAll patients have CC and MLO images for both sides
\n
This is surprising, as there seemed to be examinations in the train set where at least one of the views were missing (e.g., 5 DICOM files, 3 L-CC, 1 L-MLO and 1 R-MLO, but no R-CC).
Thank you. Is this true: 'There are new machine IDs in test dataset' ?
", "votes": 1, "replies": [ { "id": 2102062, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2023-01-16T11:09:13.710000", "content": "Yes, it is.
\nPlease check cell 10 of my notebook.
\nIt was surprising for me.
thank you.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2055387, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-05T03:52:18.257000", "content": "all zero submission, LB = ???
\nall one submission, LB = ???
\nfrom this we can compute number/ratio of pos images (for public)
All zero submission? That's just zero, right ?
", "votes": 1, "replies": [] }, { "id": 2055406, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-05T04:10:56.487000", "content": "maybe more correctly:
\nnow we \"know\" the label of the 4 test images (from your models or additional probing), then you can work out number/ratio of pos images (for public)
", "votes": 4, "replies": [] }, { "id": 2055409, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-05T04:15:59.743000", "content": "You'd have to do some shenanigans with your personal leaderboard I think to zero in on an accurate number with only two decimals to work with. Or maybe there's some way to do this in parts. Haven't really thought about it.
\nI think it's worth doing though, will look at it tomorrow. The incidence should be roughly around .02 or the split was messed with.
", "votes": 1, "replies": [] }, { "id": 2055567, "author_name": "Dieter", "author_url": "", "post_date": "2022-12-05T07:09:54.360000", "content": "submitting all ones is enough btw
\nfollow the formula
\nmean_cancer = 0.5*LB_score / (1- 0.5*LB_score)
which follows from
\nF1 = 2TP / (2TP + FN + FP)
when using that FN = 0 and FP = 1-TP in a submission of all ones
", "votes": 4, "replies": [] }, { "id": 2055591, "author_name": "Leon", "author_url": "", "post_date": "2022-12-05T07:56:22.283000", "content": "Share my result
\nall one LB: 0.04
\nmean_cancer: ~0.02
Thank you for testing!
\nConsidering the decimal rounding, LB score is 0.035-0.045, and positive rate would be 0.0178-0.0230.
\nThe value is similar to that of train set (0.0206).
i am thinking of something like this:
\nfor each machine id sort the prediction by score.
\nthreshold by score value or score ranking?
a note on threshold by score. binary threshold may not be the best. there are other way to re calibrate the prediction.
", "votes": 1, "replies": [] }, { "id": 2055836, "author_name": "", "author_url": "", "post_date": "2022-12-05T13:35:30.793000", "content": "", "votes": 0, "replies": [] }, { "id": 2060722, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-12-10T10:50:22.143000", "content": "@tomooinubushi LB decimal rounding is 0.4 = [0.0400000; 0.04999999] though
", "votes": 1, "replies": [] }, { "id": 2060840, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2022-12-10T13:54:31.177000", "content": "Thank you for pointing it out! I didn't know that.
\nSo, the positive rate is around 0.0204-0.0256, which is still similar to that of train set.
test_machine_49_count = len(test_df.query(\"machine_id == 49\"))
\ntest_len = len(test_df)
\ntest_machine_49_count/test_len > 0.40 is True. (.43 in traindf)
Thank you for your comment. It was True.
\nSo, we don't need to be afraid too much of hidden new machine IDs.
you can verify this:
\nall machine_id has at least some cancer images at hidden test
Do you mean all machines have at least one cancer image?
\nHow can I test it? Is it based on LB score values?
Yeah, I'm kinda curious how he figured this out as well. Assuming the distributions are roughly the same, you probably only have to check 190 and 197. It's possible that machine_id is simply not a signal at all and we've just got a weird split with regards to those. Not sure why'd they include it in the test data if it wasn't a signal, tbh. But I'm also confused as to why density isn't in the test set.
", "votes": 0, "replies": [] }, { "id": 2055790, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-05T12:54:25.550000", "content": "I thought he had some way of doing it without submitting for each machine id, but it occurs to me that there is some value in actually submitting for each one. It would be interesting to see if the scors line up with the traindf.
\nEg, using train
\n k train[].unique():\n (k, pfbeta(train[], [ x != k x train[]]))\n \n \n \n \n \n \n \n \n \n \nBe good to verify roughly the count of how many new machine_ids we see as well and there over all score.,
", "votes": 0, "replies": [] }, { "id": 2055814, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-05T13:10:07.027000", "content": "for 210 0, 197 0, i suspect the (few) pos cases are in the test
", "votes": 0, "replies": [] }, { "id": 2056309, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-06T01:41:50.247000", "content": "Scores don't seem to be correlating for machine_ids, so either incidence is off or distribution of machine usage.
", "votes": 0, "replies": [] } ] }, { "id": 2054476, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2022-12-04T07:16:59.677000", "content": "As with train dataset, the average site id was <1.5. I wish they were all 2;)
", "votes": 1, "replies": [ { "id": 2054524, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2022-12-04T08:01:24.697000", "content": "Thank you for your comment. I tested it, and it was true.
\nDo you think images in site id 2 are easy to diagnose?
Because the site id's for the given test were all 2.
\nI thought I could further increase the score by devising a validation set if they are all 2.
Some other distribution things to check, just to verify it was a random split and not something else:
\nIs mean age around 57.5 at site 1, 60 at site2? Maybe verify the std and nancount as well
\nAre there about 0.000644 AT views? (AT views mean a lesion was found, 10% incidence of cancer in train set)
\nImplants around 5% (often results in lower incidence of cancer due to traits of women who get implants)
I verified mean age and implant rate. Together with the positive rate from LB score, the split seems to be random.
", "votes": 0, "replies": [] }, { "id": 2055763, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-05T12:38:42.587000", "content": "Yeah, looks good!
\nI'm going to check out the implants being reported only in site 1 as well. I like how you're verifying all the natural assumptions one would make.
\nSome deeper probing on the machine ids is probably a good idea as that is a surprise.
", "votes": 0, "replies": [] } ] }, { "id": 2054654, "author_name": "MPWARE", "author_url": "", "post_date": "2022-12-04T10:20:59.223000", "content": "Thanks @tomooinubushi
\nThat's what I was wondering. Now we know better how to build our CV.
", "votes": 2, "replies": [] }, { "id": 2149352, "author_name": "sandy1112", "author_url": "", "post_date": "2023-02-18T08:00:03.670000", "content": "Apologies If I am asking a duplicate question…what is the total number of patients in the hidden test set?
", "votes": 0, "replies": [ { "id": 2149399, "author_name": "Rasoul Mojtahedzadeh", "author_url": "", "post_date": "2023-02-18T09:07:44.997000", "content": "\"You can expect roughly 8,000 patients in the hidden test set. \" - from Data section of this competition.
", "votes": 2, "replies": [ { "id": 2149439, "author_name": "sandy1112", "author_url": "", "post_date": "2023-02-18T09:51:30.327000", "content": "thanks a lot for clarifying…much appreciated
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2054249": "In [this notebook](https://www.kaggle.com/code/tomooinubushi/some-lb-probing-results-to-share/notebook), I tested some basic assumptions about test dataset.\nI demonstrated that following assumptions are all TRUE.\n\n* There are no new site ID in test dataset.\n* Patient IDs in train and test sets do not overlap\n* Image IDs in train and test sets do not overlap\n* There are no new laterality values in test dataset.\n* There are new machine IDs in test dataset. (This is already raised by @abebe9849 in [here](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369362))\n* There are no new view values in test dataset.\n* No. of images/patient are all >= 4\n* Site ID is always the same for each patient.\n* Age is always the same for each patient.\n* There are no overlap of machine IDs between two sites in test dataset.\n* Some patients underwent mammography with multiple machines.\n\n[updated at 2022-12-04]\n* No. of images in site ID 1 > No. of images in site ID 2. (by @yujiariyasu)\n\n[updated at 2022-12-05]\n* All patients have CC and MLO images for both sides\n* More than 40% of images are from machine ID 49 (43% for train dataset) (by @kaggleqrdl)\n* Mean age of patients is between 56-61 (58.6 for train set), and patients in site 1 are younger than those in site 2. (by @kaggleqrdl)\n* 1-2% of patients use implants (1.4% for train set).(by @kaggleqrdl)\n* **Positive rate is between 0.0204-0.0256** (0.0206 for train dataset). (shown by @zzy990106 and others)\n\n[updated at 2022-12-15]\n* Age column contains nan, while others do not\n\nI am happy if anyone correct me if I am wrong.\nI am also very happy if anyone share us other assumptions/hypothesis about test dataset.", "2148868": ">All patients have CC and MLO images for both sides\n\nThis is surprising, as there seemed to be examinations in the **train set** where at least one of the views were missing (e.g., 5 `DICOM` files, 3 L-CC, 1 L-MLO and 1 R-MLO, but no R-CC).", "2101738": "Thank you. Is this true: '*There are new machine IDs in test dataset'* ?", "2055387": "all zero submission, LB = ???\nall one submission, LB = ???\nfrom this we can compute number/ratio of pos images (for public)", "2055362": "test_machine_49_count = len(test_df.query(\"machine_id == 49\"))\ntest_len = len(test_df)\ntest_machine_49_count/test_len > 0.40 is True. (.43 in traindf)\n", "2054484": "you can verify this:\nall machine\\_id has at least some cancer images at hidden test", "2054476": "As with train dataset, the average site id was <1.5. I wish they were all 2;)", "2055424": "Some other distribution things to check, just to verify it was a random split and not something else:\n\nIs mean age around 57.5 at site 1, 60 at site2? Maybe verify the std and nancount as well\nAre there about 0.000644 AT views? (AT views mean a lesion was found, 10% incidence of cancer in train set)\nImplants around 5% (often results in lower incidence of cancer due to traits of women who get implants)\n\n", "2054654": "Thanks @tomooinubushi \n\n- Patient IDs in train and test sets do not overlap\n- Image IDs in train and test sets do not overlap\n\nThat's what I was wondering. Now we know better how to build our CV.\n", "2149352": "Apologies If I am asking a duplicate question...what is the total number of patients in the hidden test set?" } }