@sohier dropped this nugget on one of the threads
\n\n\nHowever, the RSNA team invested a lot of effort in enriching the competition dataset for cancer cases to ensure there are enough of them to use for modeling. From memory, the cancer rate in the general population is closer to 0.1% (@vaillant might have a more accurate number). Since the false negatives weren't enriched there's a reasonable chance that there's literally only one in the entire dataset.
\n
https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369262#2056107
\nIndeed, I noticed a statistical curiosity. Could be coincidence, but if not certainly would point to an enriched dataset.
\ngb_lat = origtrain.drop_duplicates([, ]).groupby([, , ]).count()[]\ndisplay(gb_lat)\nlaterality site_id cancer\nL \n \n \n \nR \n \n \n \nName: age, dtype: int64\nMy reading of this is that site 2 has the same amount of Left and Right laterality incidences of cancer. It could be total coincidence, of course, but it could also be an example of how the data managers wanted to make sure there were sufficient examples of both laterality with cancer.
\nLiterature is a bit conflicted on this, but some reports say that left breasts have a 5-10% greater chance of cancer. The question of course is how enriched the hidden set is.
", "messages": [ { "id": 2058352, "postDate": "2022-12-07T20:45:01.567Z", "content": "@sohier dropped this nugget on one of the threads
\n\n\nHowever, the RSNA team invested a lot of effort in enriching the competition dataset for cancer cases to ensure there are enough of them to use for modeling. From memory, the cancer rate in the general population is closer to 0.1% (@vaillant might have a more accurate number). Since the false negatives weren't enriched there's a reasonable chance that there's literally only one in the entire dataset.
\n
https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369262#2056107
\nIndeed, I noticed a statistical curiosity. Could be coincidence, but if not certainly would point to an enriched dataset.
\ngb_lat = origtrain.drop_duplicates([, ]).groupby([, , ]).count()[]\ndisplay(gb_lat)\nlaterality site_id cancer\nL \n \n \n \nR \n \n \n \nName: age, dtype: int64\nMy reading of this is that site 2 has the same amount of Left and Right laterality incidences of cancer. It could be total coincidence, of course, but it could also be an example of how the data managers wanted to make sure there were sufficient examples of both laterality with cancer.
\nLiterature is a bit conflicted on this, but some reports say that left breasts have a 5-10% greater chance of cancer. The question of course is how enriched the hidden set is.
", "rawMarkdown": "@sohier dropped this nugget on one of the threads\n\n> However, the RSNA team invested a lot of effort in enriching the competition dataset for cancer cases to ensure there are enough of them to use for modeling. From memory, the cancer rate in the general population is closer to 0.1% (@vaillant might have a more accurate number). Since the false negatives weren't enriched there's a reasonable chance that there's literally only one in the entire dataset.\n\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369262#2056107\n\nIndeed, I noticed a statistical curiosity. Could be coincidence, but if not certainly would point to an enriched dataset.\n\n```python\ngb_lat = origtrain.drop_duplicates(['patient_id', 'laterality']).groupby([\"laterality\", \"site_id\", 'cancer']).count()['age']\ndisplay(gb_lat)\n```\n\n```python\nlaterality site_id cancer\nL 1 0 5681\n 1 129\n 2 0 5976\n 1 119\nR 1 0 5685\n 1 125\n 2 0 5976\n 1 119\nName: age, dtype: int64\n```\n\nMy reading of this is that site 2 has the same amount of Left and Right laterality incidences of cancer. It could be total coincidence, of course, but it could also be an example of how the data managers wanted to make sure there were sufficient examples of both laterality with cancer. \n\nLiterature is a bit conflicted on this, but some reports say that left breasts have a 5-10% greater chance of cancer. The question of course is how enriched the hidden set is.", "votes": 6 } ], "comments": [], "raw_markdown_by_id": { "2058352": "@sohier dropped this nugget on one of the threads\n\n> However, the RSNA team invested a lot of effort in enriching the competition dataset for cancer cases to ensure there are enough of them to use for modeling. From memory, the cancer rate in the general population is closer to 0.1% (@vaillant might have a more accurate number). Since the false negatives weren't enriched there's a reasonable chance that there's literally only one in the entire dataset.\n\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369262#2056107\n\nIndeed, I noticed a statistical curiosity. Could be coincidence, but if not certainly would point to an enriched dataset.\n\n```python\ngb_lat = origtrain.drop_duplicates(['patient_id', 'laterality']).groupby([\"laterality\", \"site_id\", 'cancer']).count()['age']\ndisplay(gb_lat)\n```\n\n```python\nlaterality site_id cancer\nL 1 0 5681\n 1 129\n 2 0 5976\n 1 119\nR 1 0 5685\n 1 125\n 2 0 5976\n 1 119\nName: age, dtype: int64\n```\n\nMy reading of this is that site 2 has the same amount of Left and Right laterality incidences of cancer. It could be total coincidence, of course, but it could also be an example of how the data managers wanted to make sure there were sufficient examples of both laterality with cancer. \n\nLiterature is a bit conflicted on this, but some reports say that left breasts have a 5-10% greater chance of cancer. The question of course is how enriched the hidden set is." } }