How well do solutions from Kaggle competitions translate to new data? Join our effort to find out.
\nRSNA is planning to conduct follow-on research to analyze the performance of models created for the challenge against a new dataset. This project will not affect the results of the challenge or prizes awarded. We will run the winners’ models (top 8) against this new dataset, and invite other competitors to submit their models for this project, as well.
\nWe will launch the post hoc analysis project after validation and announcement of winners from the challenge. We invite competitors interested in participating to complete the survey linked here by April 10 and to submit the following:
\nThe data used in the challenge competition was provided by two sources (from the US and Australia). The post hoc analysis will use the PERFORMS dataset, which contains data from 86 sites and is maintained for the external quality assurance of breast screening readers in the UK NHS Breast Screening Programme.
\nThe goal of the post hoc analysis is to assess how well challenge models generalize to this expertly curated dataset, and how the performance of these algorithms compare to thousands of breast screening readers.
\nThe results will be scored using the same metric used for the challenge competition (and potentially other metrics) and will be shared with the teams who created the models. Participating teams will have the option of submitting one model modified to run more effectively against unseen data.
\nThe RSNA task force plans to author one or more research papers based on this work, which will include comparative analyses of the scores from the competition and those generated in the post hoc analysis. Competitors whose models are used in the post hoc analysis will be invited to participate as contributors to at least one such research publication.
", "messages": [ { "id": 2154314, "postDate": "2023-02-21T23:56:46.007Z", "content": "How well do solutions from Kaggle competitions translate to new data? Join our effort to find out.
\nRSNA is planning to conduct follow-on research to analyze the performance of models created for the challenge against a new dataset. This project will not affect the results of the challenge or prizes awarded. We will run the winners’ models (top 8) against this new dataset, and invite other competitors to submit their models for this project, as well.
\nWe will launch the post hoc analysis project after validation and announcement of winners from the challenge. We invite competitors interested in participating to complete the survey linked here by April 10 and to submit the following:
\nThe data used in the challenge competition was provided by two sources (from the US and Australia). The post hoc analysis will use the PERFORMS dataset, which contains data from 86 sites and is maintained for the external quality assurance of breast screening readers in the UK NHS Breast Screening Programme.
\nThe goal of the post hoc analysis is to assess how well challenge models generalize to this expertly curated dataset, and how the performance of these algorithms compare to thousands of breast screening readers.
\nThe results will be scored using the same metric used for the challenge competition (and potentially other metrics) and will be shared with the teams who created the models. Participating teams will have the option of submitting one model modified to run more effectively against unseen data.
\nThe RSNA task force plans to author one or more research papers based on this work, which will include comparative analyses of the scores from the competition and those generated in the post hoc analysis. Competitors whose models are used in the post hoc analysis will be invited to participate as contributors to at least one such research publication.
", "rawMarkdown": "How well do solutions from Kaggle competitions translate to new data? Join our effort to find out. \n\nRSNA is planning to conduct follow-on research to analyze the performance of models created for the challenge against a new dataset. This project will not affect the results of the challenge or prizes awarded. We will run the winners’ models (top 8) against this new dataset, and invite other competitors to submit their models for this project, as well. \n\nWe will launch the post hoc analysis project after validation and announcement of winners from the challenge. We invite competitors interested in participating to [complete the survey linked here by April 10](https://forms.gle/P7YzV6We6U2tEYfGA) and to submit the following: \n\n- code used to train the model\n- solution write-up (at least one detailed paragraph). Include all fields the model accepts with details about any hard-coded fields.\n- a generic inference notebook that can be applied to unseen data (Note: site_id and vendor_id will not be present in the new dataset.)\n\nThe data used in the challenge competition was provided by two sources (from the US and Australia). The post hoc analysis will use the PERFORMS dataset, which contains data from 86 sites and is maintained for the external quality assurance of breast screening readers in the UK NHS Breast Screening Programme.\n\nThe goal of the post hoc analysis is to assess how well challenge models generalize to this expertly curated dataset, and how the performance of these algorithms compare to thousands of breast screening readers.\n\nThe results will be scored using the same metric used for the challenge competition (and potentially other metrics) and will be shared with the teams who created the models. Participating teams will have the option of submitting one model modified to run more effectively against unseen data.\n\nThe RSNA task force plans to author one or more research papers based on this work, which will include comparative analyses of the scores from the competition and those generated in the post hoc analysis. Competitors whose models are used in the post hoc analysis will be invited to participate as contributors to at least one such research publication. \n", "votes": 31 }, { "id": 2165997, "postDate": "2023-03-02T15:15:06.597Z", "content": "I'm getting permission denied on the link. Would it be possible to check this ? Also, do winning teams need to fill it in, or you will contact us in any case.
\nYou need permission\nThis form can only be viewed by users in the owner's organization.\n\nTry contacting the owner of the form if you think this is a mistake. Learn More.\nThanks for flagging this. I'll work with Chris to get that fixed.
", "rawMarkdown": "Thanks for flagging this. I'll work with Chris to get that fixed.", "votes": 1, "replies": [ { "id": 2171629, "postDate": "2023-03-07T00:31:33.343Z", "content": "Sorry all, I think I have opened up permissions on the form so you should be able to post. Please post if it isn't working. Thanks! Again, the link.
", "rawMarkdown": "Sorry all, I think I have opened up permissions on the form so you should be able to post. Please post if it isn't working. Thanks! Again, [the link](https://docs.google.com/forms/d/e/1FAIpQLSf3QwjWgQb0GHywsU9PGP-meHAEv0fEcXskbIdgop4j5PNhkQ/viewform?usp=sharing).", "votes": 2 }, { "id": 2171634, "postDate": "2023-03-07T00:38:50.257Z", "content": "Yes, it's working, just checked.
", "rawMarkdown": "Yes, it's working, just checked. ", "votes": 1 }, { "id": 2176126, "postDate": "2023-03-10T12:19:22.973Z", "content": "Thanks, 🙏 it's working now, just submitted.
", "rawMarkdown": "Thanks, 🙏 it's working now, just submitted. " } ] } ] }, { "id": 2180415, "postDate": "2023-03-13T19:28:35.627Z", "content": "Hi @cdcarr , @sohier
\nI want to ask if its possible to release the test set. I looked a lot into GPU-based decoding of the DICOMs and although it worked fine for all train images in the end, the decode failed for a few test images. I would like to figure out what the problem was and improve DALI/ nvjpeg with that, but cant debug without access to the test data.
I second this @cdcarr and @sohier would this be possible? That would be awesome.
", "rawMarkdown": "I second this @cdcarr and @sohier would this be possible? That would be awesome." }, { "id": 2202275, "postDate": "2023-03-29T21:54:26.957Z", "content": "Unfortunately the answer is no.
", "rawMarkdown": "Unfortunately the answer is no.", "votes": 2, "replies": [ { "id": 2203835, "postDate": "2023-03-31T06:57:45.583Z", "content": "thanks for letting us know
", "rawMarkdown": "thanks for letting us know", "votes": 1 } ] } ] }, { "id": 2170377, "postDate": "2023-03-06T00:05:57.373Z", "content": "Super glad you're doing this. It's something that should be the primary focus on kaggle, tbh. Overfitting and lack of generalizability is a serious issue which continuously gets ignored on Kaggle for reasons I truly do not understand.
\nHonestly, I'd love to see some comps, even if the rewards are less and perhaps no medals, on simply predicting / evaluating models and how well they will generalize. Eg: You get the model and some data, and the one who creates the best way to evaluate / cross validate wins.
\nMaybe they're not medal comps, and maybe invite only, but still. We need to put more focus on this.
\nOne of the great outcomes of these types of comps would be deep analysis of winning models, which there really needs to be more of as well.
", "rawMarkdown": "Super glad you're doing this. It's something that should be the primary focus on kaggle, tbh. Overfitting and lack of generalizability is a serious issue which continuously gets ignored on Kaggle for reasons I truly do not understand.\n\nHonestly, I'd love to see some comps, even if the rewards are less and perhaps no medals, on simply predicting / evaluating models and how well they will generalize. Eg: You get the model and some data, and the one who creates the best way to evaluate / cross validate wins.\n\nMaybe they're not medal comps, and maybe invite only, but still. We need to put more focus on this.\n\nOne of the great outcomes of these types of comps would be deep analysis of winning models, which there really needs to be more of as well.\n\n\n" }, { "id": 2163702, "postDate": "2023-03-01T02:32:06.897Z", "content": "Only the invited competitor can participate in this project?
", "rawMarkdown": "Only the invited competitor can participate in this project?", "replies": [ { "id": 2164601, "postDate": "2023-03-01T17:29:01.123Z", "content": "Good question! Any competitor can request to participate by completing the form. The planning task force will select a group from interested candidates based on the resources we have available for the project.
", "rawMarkdown": "Good question! Any competitor can request to participate by completing the form. The planning task force will select a group from interested candidates based on the resources we have available for the project.", "votes": 1 } ] } ], "comments": [ { "id": 2165997, "author_name": "Darragh", "author_url": "", "post_date": "2023-03-02T15:15:06.597000", "content": "I'm getting permission denied on the link. Would it be possible to check this ? Also, do winning teams need to fill it in, or you will contact us in any case.
\nYou need permission\nThis form can only be viewed by users in the owner's organization.\n\nTry contacting the owner of the form if you think this is a mistake. Learn More.\nThanks for flagging this. I'll work with Chris to get that fixed.
", "votes": 1, "replies": [ { "id": 2171629, "author_name": "Chris Carr", "author_url": "", "post_date": "2023-03-07T00:31:33.343000", "content": "Sorry all, I think I have opened up permissions on the form so you should be able to post. Please post if it isn't working. Thanks! Again, the link.
", "votes": 2, "replies": [] }, { "id": 2171634, "author_name": "Maggie", "author_url": "", "post_date": "2023-03-07T00:38:50.257000", "content": "Yes, it's working, just checked.
", "votes": 1, "replies": [] }, { "id": 2176126, "author_name": "Darragh", "author_url": "", "post_date": "2023-03-10T12:19:22.973000", "content": "Thanks, 🙏 it's working now, just submitted.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2180415, "author_name": "Dieter", "author_url": "", "post_date": "2023-03-13T19:28:35.627000", "content": "Hi @cdcarr , @sohier
\nI want to ask if its possible to release the test set. I looked a lot into GPU-based decoding of the DICOMs and although it worked fine for all train images in the end, the decode failed for a few test images. I would like to figure out what the problem was and improve DALI/ nvjpeg with that, but cant debug without access to the test data.
I second this @cdcarr and @sohier would this be possible? That would be awesome.
", "votes": 0, "replies": [] }, { "id": 2202275, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-03-29T21:54:26.957000", "content": "Unfortunately the answer is no.
", "votes": 2, "replies": [ { "id": 2203835, "author_name": "Dieter", "author_url": "", "post_date": "2023-03-31T06:57:45.583000", "content": "thanks for letting us know
", "votes": 1, "replies": [] } ] } ] }, { "id": 2170377, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-03-06T00:05:57.373000", "content": "Super glad you're doing this. It's something that should be the primary focus on kaggle, tbh. Overfitting and lack of generalizability is a serious issue which continuously gets ignored on Kaggle for reasons I truly do not understand.
\nHonestly, I'd love to see some comps, even if the rewards are less and perhaps no medals, on simply predicting / evaluating models and how well they will generalize. Eg: You get the model and some data, and the one who creates the best way to evaluate / cross validate wins.
\nMaybe they're not medal comps, and maybe invite only, but still. We need to put more focus on this.
\nOne of the great outcomes of these types of comps would be deep analysis of winning models, which there really needs to be more of as well.
", "votes": 0, "replies": [] }, { "id": 2163702, "author_name": "Lau2664", "author_url": "", "post_date": "2023-03-01T02:32:06.897000", "content": "Only the invited competitor can participate in this project?
", "votes": 0, "replies": [ { "id": 2164601, "author_name": "Chris Carr", "author_url": "", "post_date": "2023-03-01T17:29:01.123000", "content": "Good question! Any competitor can request to participate by completing the form. The planning task force will select a group from interested candidates based on the resources we have available for the project.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2154314": "How well do solutions from Kaggle competitions translate to new data? Join our effort to find out. \n\nRSNA is planning to conduct follow-on research to analyze the performance of models created for the challenge against a new dataset. This project will not affect the results of the challenge or prizes awarded. We will run the winners’ models (top 8) against this new dataset, and invite other competitors to submit their models for this project, as well. \n\nWe will launch the post hoc analysis project after validation and announcement of winners from the challenge. We invite competitors interested in participating to [complete the survey linked here by April 10](https://forms.gle/P7YzV6We6U2tEYfGA) and to submit the following: \n\n- code used to train the model\n- solution write-up (at least one detailed paragraph). Include all fields the model accepts with details about any hard-coded fields.\n- a generic inference notebook that can be applied to unseen data (Note: site_id and vendor_id will not be present in the new dataset.)\n\nThe data used in the challenge competition was provided by two sources (from the US and Australia). The post hoc analysis will use the PERFORMS dataset, which contains data from 86 sites and is maintained for the external quality assurance of breast screening readers in the UK NHS Breast Screening Programme.\n\nThe goal of the post hoc analysis is to assess how well challenge models generalize to this expertly curated dataset, and how the performance of these algorithms compare to thousands of breast screening readers.\n\nThe results will be scored using the same metric used for the challenge competition (and potentially other metrics) and will be shared with the teams who created the models. Participating teams will have the option of submitting one model modified to run more effectively against unseen data.\n\nThe RSNA task force plans to author one or more research papers based on this work, which will include comparative analyses of the scores from the competition and those generated in the post hoc analysis. Competitors whose models are used in the post hoc analysis will be invited to participate as contributors to at least one such research publication. \n", "2165997": "I'm getting permission denied on the [link](https://forms.gle/P7YzV6We6U2tEYfGA). Would it be possible to check this ? Also, do winning teams need to fill it in, or you will contact us in any case. \n```\nYou need permission\nThis form can only be viewed by users in the owner's organization.\n\nTry contacting the owner of the form if you think this is a mistake. Learn More.\n```", "2180415": "Hi @cdcarr , @sohier\nI want to ask if its possible to release the test set. I looked a lot into GPU-based decoding of the DICOMs and although it worked fine for all train images in the end, the decode failed for a few test images. I would like to figure out what the problem was and improve DALI/ nvjpeg with that, but cant debug without access to the test data. ", "2170377": "Super glad you're doing this. It's something that should be the primary focus on kaggle, tbh. Overfitting and lack of generalizability is a serious issue which continuously gets ignored on Kaggle for reasons I truly do not understand.\n\nHonestly, I'd love to see some comps, even if the rewards are less and perhaps no medals, on simply predicting / evaluating models and how well they will generalize. Eg: You get the model and some data, and the one who creates the best way to evaluate / cross validate wins.\n\nMaybe they're not medal comps, and maybe invite only, but still. We need to put more focus on this.\n\nOne of the great outcomes of these types of comps would be deep analysis of winning models, which there really needs to be more of as well.\n\n\n", "2163702": "Only the invited competitor can participate in this project?" } }