@guntasdhanjal need some clarification about “no external data”
\n1.Pretraining on non-medical datasets
\nAre models that are pretrained on generic datasets like ImageNet (e.g., EfficientNet, ConvNeXt, ViT, ResNet) allowed? These models are trained on natural images, not medical images.
2.Pretraining on medical datasets
\nAre models that are pretrained on medical imaging datasets (e.g., chest X-ray foundation models like EVA-X, CheXZero, or similar) allowed? These have been trained directly on chest X-ray data outside of the competition dataset.
Could you please confirm whether any(or both) of these cases fall under the “external data” restriction?
", "messages": [ { "id": 3286069, "postDate": "2025-09-09T07:49:03.053Z", "content": "I'd like to argue that EVA-X should be banned from the competition: It is contradictory that data from division B must not be used for training division A models whereas EVA-X is trained on a superset of that data.
\nFurthermore, we can question whether it matters that EVA-X's training data include the test dataset of this competition: The golden rule of data science stipulates that models must be always evaluated on held-out data. Scoring a model on its own training data is meaningless and doesn't say anything about its generalization ability. The authors of EVA-X were aware of the issue and in their own experiments applied the principle: We do not use any of the images tested subsequently for training, even though they are unlabeled. Even if a model derived from EVA-X scores high on the private leaderboard, we cannot be sure that it will generalize to unseen data.
\nFor those who want to learn more about this interesting model: EVA-X: A foundation model for general chest X-ray analysis with self-supervised learning
", "rawMarkdown": "I'd like to argue that EVA-X should be banned from the competition: It is contradictory that data from division B must not be used for training division A models whereas EVA-X is trained on a superset of that data.\n\nFurthermore, we can question whether it matters that EVA-X's training data include the test dataset of this competition: The golden rule of data science stipulates that models must be always evaluated on held-out data. Scoring a model on its own training data is meaningless and doesn't say anything about its generalization ability. The authors of EVA-X were aware of the issue and in their own experiments applied the principle: *We do not use any of the images tested subsequently for training, even though they are unlabeled.* Even if a model derived from EVA-X scores high on the private leaderboard, we cannot be sure that it will generalize to unseen data.\n\nFor those who want to learn more about this interesting model: [EVA-X: A foundation model for general chest X-ray analysis with self-supervised learning](https://arxiv.org/pdf/2405.05237)", "votes": 1, "replies": [ { "id": 3286092, "postDate": "2025-09-09T08:19:29.920Z", "content": "@ambrosm I found a CheXpert dataset on Kaggle, but the labels seem to be missing. Could it be that the dataset in this competition is the labeled version of it?
", "rawMarkdown": "@ambrosm I found a CheXpert dataset on Kaggle, but the labels seem to be missing. Could it be that the dataset in this competition is the labeled version of it?", "replies": [ { "id": 3286113, "postDate": "2025-09-09T08:41:25.293Z", "content": "Hi @nguyncdngs, a search for CheXpert datasets on Kaggle returns 62 datasets. Many of them contain 223649 images. Some of them contain labels. Image resolution varies. Some contain no images but models.
\nIf I visually compare test1/00000005_001_001.jpg of this competition with train/patient00005/study1/view1_frontal.jpg of https://www.kaggle.com/datasets/ashery/chexpert, they look very very similar. The dataset contains labels.
", "rawMarkdown": "Hi @nguyncdngs, a search for [CheXpert datasets on Kaggle](https://www.kaggle.com/datasets?search=CheXpert) returns 62 datasets. Many of them contain 223649 images. Some of them contain labels. Image resolution varies. Some contain no images but models.\n\nIf I visually compare test1/00000005_001_001.jpg of this competition with train/patient00005/study1/view1_frontal.jpg of https://www.kaggle.com/datasets/ashery/chexpert, they look very very similar. The dataset contains labels." }, { "id": 3286118, "postDate": "2025-09-09T08:51:19.303Z", "content": "@ambrosm I also found that dataset, but it contains 1, 0, -1, and missing values. The missing values may not represent 0, so I wonder how the host can annotate them.
", "rawMarkdown": "@ambrosm I also found that dataset, but it contains 1, 0, -1, and missing values. The missing values may not represent 0, so I wonder how the host can annotate them." } ] }, { "id": 3286164, "postDate": "2025-09-09T10:41:41.233Z", "content": "I came across a new model called CXformer(S), which shows comparable performance to Eva_X. Do you think this model could be applied here?
", "rawMarkdown": "I came across a new model called [CXformer(S)](https://github.com/m42-health/CXformer), which shows comparable performance to Eva_X. Do you think this model could be applied here?", "replies": [ { "id": 3286167, "postDate": "2025-09-09T10:46:12.613Z", "content": "Also, from my recent experiments, just changing the pretrained model gives only a tiny improvement (~0.1%). To meaningfully boost the LB score, we need more effort analyzing the raw data and extracting actionable insights.
", "rawMarkdown": "Also, from my recent experiments, just changing the pretrained model gives only a tiny improvement (~0.1%). To meaningfully boost the LB score, we need more effort analyzing the raw data and extracting actionable insights." } ] } ] }, { "id": 3286038, "postDate": "2025-09-09T06:13:40.480Z", "content": "@guntasdhanjal need some clarification about “no external data”
\n1.Pretraining on non-medical datasets
\nAre models that are pretrained on generic datasets like ImageNet (e.g., EfficientNet, ConvNeXt, ViT, ResNet) allowed? These models are trained on natural images, not medical images.
2.Pretraining on medical datasets
\nAre models that are pretrained on medical imaging datasets (e.g., chest X-ray foundation models like EVA-X, CheXZero, or similar) allowed? These have been trained directly on chest X-ray data outside of the competition dataset.
Could you please confirm whether any(or both) of these cases fall under the “external data” restriction?
", "rawMarkdown": "@guntasdhanjal need some clarification about “no external data”\n\n1.Pretraining on non-medical datasets\nAre models that are pretrained on generic datasets like ImageNet (e.g., EfficientNet, ConvNeXt, ViT, ResNet) allowed? These models are trained on natural images, not medical images.\n\n2.Pretraining on medical datasets\nAre models that are pretrained on medical imaging datasets (e.g., chest X-ray foundation models like EVA-X, CheXZero, or similar) allowed? These have been trained directly on chest X-ray data outside of the competition dataset.\n\nCould you please confirm whether any(or both) of these cases fall under the “external data” restriction?\n", "votes": 2 }, { "id": 3286120, "postDate": "2025-09-09T09:02:26.493Z", "content": "Thanks @adarsh1403 for the great question! Here’s the clarification:
\nPretraining on generic datasets (like ImageNet) is allowed. Transfer learning from non-medical sources is standard practice and not considered “external data.”
Pretraining on publicly available medical datasets (e.g., EVA-X, CheXZero, CheXFound, etc.) is also allowed, since these models are openly accessible to all participants. The key requirement is that your final fine-tuning and submission must be done inside Kaggle so results remain fair and reproducible.
According to this clarification, the following would be allowed:
\nIn both cases, the final fine-tuning and submission can be done inside Kaggle.
", "rawMarkdown": "According to this clarification, the following would be allowed:\n- Training on Division B data because the dataset is publicly available and openly accessible to all participants\n- Training on https://www.kaggle.com/datasets/ashery/chexpert because the dataset is publicly available and openly accessible to all participants\n\nIn both cases, the final fine-tuning and submission can be done inside Kaggle.\n", "replies": [ { "id": 3286134, "postDate": "2025-09-09T09:31:40.353Z", "content": "Thanks @ambrosm for raising this point. Just to clarify: participants may use external datasets or foundation models only if they are publicly available and equally accessible to all participants at no cost. Regardless of the source, all fine-tuning and final submissions must be done inside Kaggle to ensure fairness and reproducibility.
\nWe encourage participants to follow this guideline strictly, and any questions about the suitability of specific external datasets can be asked in the forums for clarification.
", "rawMarkdown": "Thanks @ambrosm for raising this point. Just to clarify: participants may use external datasets or foundation models only if they are publicly available and equally accessible to all participants at no cost. Regardless of the source, all fine-tuning and final submissions must be done inside Kaggle to ensure fairness and reproducibility.\n\nWe encourage participants to follow this guideline strictly, and any questions about the suitability of specific external datasets can be asked in the forums for clarification." } ] }, { "id": 3286133, "postDate": "2025-09-09T09:27:51.917Z", "content": "In an earlier discussion, you mentioned that “The full dataset is over 1TB, which can be a barrier for people without high-end resources (like large storage or paid GPU access). By creating two divisions, we give more participants the chance to compete — even those who want to work with lighter setups.”
\nThis made sense because it ensured fairness for participants with limited compute. However, allowing pretrained medical models (which themselves were trained on millions of X-rays) feels contradictory to your previous statement.
Also,since many foundation models (EVA-X, CheXZero, CheXFound, etc.) are trained on huge public datasets, what are the chances that the competition’s test set may already be part of their pretraining data(as stated by @ambrosm )? If that’s possible, wouldn’t it raise concerns about data leakage or unfair advantage?
", "rawMarkdown": "In an earlier discussion, you mentioned that “The full dataset is over 1TB, which can be a barrier for people without high-end resources (like large storage or paid GPU access). By creating two divisions, we give more participants the chance to compete — even those who want to work with lighter setups.”\nThis made sense because it ensured fairness for participants with limited compute. However, allowing pretrained medical models (which themselves were trained on millions of X-rays) feels contradictory to your previous statement.\n\nAlso,since many foundation models (EVA-X, CheXZero, CheXFound, etc.) are trained on huge public datasets, what are the chances that the competition’s test set may already be part of their pretraining data(as stated by @ambrosm )? If that’s possible, wouldn’t it raise concerns about data leakage or unfair advantage?\n", "replies": [ { "id": 3286137, "postDate": "2025-09-09T09:39:05.023Z", "content": "Thanks @adarsh1403 for the thoughtful question.
\nWe understand the concern regarding pretrained medical models and potential overlap with unseen test data. The key principle is that all fine-tuning and submissions must be done inside Kaggle, ensuring that every participant works under the same conditions and that results remain reproducible.
\nWhile foundation models may have been trained on large public datasets, participants must comply with the rule that only publicly accessible data and models are used, and no private or competition-specific test data may be incorporated during training. This approach maintains fairness, even when pretrained models are leveraged.
\nAny questions about whether a specific model or dataset is allowed can be asked in the forums to ensure clarity for all participants.
", "rawMarkdown": "Thanks @adarsh1403 for the thoughtful question.\n\nWe understand the concern regarding pretrained medical models and potential overlap with unseen test data. The key principle is that all fine-tuning and submissions must be done inside Kaggle, ensuring that every participant works under the same conditions and that results remain reproducible.\n\nWhile foundation models may have been trained on large public datasets, participants must comply with the rule that only publicly accessible data and models are used, and no private or competition-specific test data may be incorporated during training. This approach maintains fairness, even when pretrained models are leveraged.\n\nAny questions about whether a specific model or dataset is allowed can be asked in the forums to ensure clarity for all participants.", "votes": -1 }, { "id": 3300861, "postDate": "2025-10-11T19:40:22.627Z", "content": "@guntasdhanjal I would like to know if it is permissible to use Division A data for training a model intended for Division B
", "rawMarkdown": "@guntasdhanjal I would like to know if it is permissible to use Division A data for training a model intended for Division B" }, { "id": 3301380, "postDate": "2025-10-13T08:48:55.717Z", "content": "Hi @sonnguyenk17 ,
\nEach division has its own independent dataset — the images in Division A and Division B are completely different.
\nTo ensure fairness, models for Division B should be trained only on Division B’s training data.
You can, of course, reuse your workflow or architecture between divisions, but data from one division shouldn’t be used to train models for the other.
", "rawMarkdown": "Hi @sonnguyenk17 ,\n\nEach division has its own independent dataset — the images in Division A and Division B are completely different.\nTo ensure fairness, models for Division B should be trained only on Division B’s training data.\n\nYou can, of course, reuse your workflow or architecture between divisions, but data from one division shouldn’t be used to train models for the other." } ] } ] } ], "comments": [ { "id": 3286069, "author_name": "AmbrosM", "author_url": "", "post_date": "2025-09-09T07:49:03.053000", "content": "I'd like to argue that EVA-X should be banned from the competition: It is contradictory that data from division B must not be used for training division A models whereas EVA-X is trained on a superset of that data.
\nFurthermore, we can question whether it matters that EVA-X's training data include the test dataset of this competition: The golden rule of data science stipulates that models must be always evaluated on held-out data. Scoring a model on its own training data is meaningless and doesn't say anything about its generalization ability. The authors of EVA-X were aware of the issue and in their own experiments applied the principle: We do not use any of the images tested subsequently for training, even though they are unlabeled. Even if a model derived from EVA-X scores high on the private leaderboard, we cannot be sure that it will generalize to unseen data.
\nFor those who want to learn more about this interesting model: EVA-X: A foundation model for general chest X-ray analysis with self-supervised learning
", "votes": 1, "replies": [ { "id": 3286092, "author_name": "Duong Nguyen", "author_url": "", "post_date": "2025-09-09T08:19:29.920000", "content": "@ambrosm I found a CheXpert dataset on Kaggle, but the labels seem to be missing. Could it be that the dataset in this competition is the labeled version of it?
", "votes": 0, "replies": [ { "id": 3286113, "author_name": "AmbrosM", "author_url": "", "post_date": "2025-09-09T08:41:25.293000", "content": "Hi @nguyncdngs, a search for CheXpert datasets on Kaggle returns 62 datasets. Many of them contain 223649 images. Some of them contain labels. Image resolution varies. Some contain no images but models.
\nIf I visually compare test1/00000005_001_001.jpg of this competition with train/patient00005/study1/view1_frontal.jpg of https://www.kaggle.com/datasets/ashery/chexpert, they look very very similar. The dataset contains labels.
", "votes": 0, "replies": [] }, { "id": 3286118, "author_name": "Duong Nguyen", "author_url": "", "post_date": "2025-09-09T08:51:19.303000", "content": "@ambrosm I also found that dataset, but it contains 1, 0, -1, and missing values. The missing values may not represent 0, so I wonder how the host can annotate them.
", "votes": 0, "replies": [] } ] }, { "id": 3286164, "author_name": "AnnieGo", "author_url": "", "post_date": "2025-09-09T10:41:41.233000", "content": "I came across a new model called CXformer(S), which shows comparable performance to Eva_X. Do you think this model could be applied here?
", "votes": 0, "replies": [ { "id": 3286167, "author_name": "AnnieGo", "author_url": "", "post_date": "2025-09-09T10:46:12.613000", "content": "Also, from my recent experiments, just changing the pretrained model gives only a tiny improvement (~0.1%). To meaningfully boost the LB score, we need more effort analyzing the raw data and extracting actionable insights.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3286120, "author_name": "Guntas Dhanjal", "author_url": "", "post_date": "2025-09-09T09:02:26.493000", "content": "Thanks @adarsh1403 for the great question! Here’s the clarification:
\nPretraining on generic datasets (like ImageNet) is allowed. Transfer learning from non-medical sources is standard practice and not considered “external data.”
Pretraining on publicly available medical datasets (e.g., EVA-X, CheXZero, CheXFound, etc.) is also allowed, since these models are openly accessible to all participants. The key requirement is that your final fine-tuning and submission must be done inside Kaggle so results remain fair and reproducible.
According to this clarification, the following would be allowed:
\nIn both cases, the final fine-tuning and submission can be done inside Kaggle.
", "votes": 0, "replies": [ { "id": 3286134, "author_name": "Guntas Dhanjal", "author_url": "", "post_date": "2025-09-09T09:31:40.353000", "content": "Thanks @ambrosm for raising this point. Just to clarify: participants may use external datasets or foundation models only if they are publicly available and equally accessible to all participants at no cost. Regardless of the source, all fine-tuning and final submissions must be done inside Kaggle to ensure fairness and reproducibility.
\nWe encourage participants to follow this guideline strictly, and any questions about the suitability of specific external datasets can be asked in the forums for clarification.
", "votes": 0, "replies": [] } ] }, { "id": 3286133, "author_name": "adarsh1403", "author_url": "", "post_date": "2025-09-09T09:27:51.917000", "content": "In an earlier discussion, you mentioned that “The full dataset is over 1TB, which can be a barrier for people without high-end resources (like large storage or paid GPU access). By creating two divisions, we give more participants the chance to compete — even those who want to work with lighter setups.”
\nThis made sense because it ensured fairness for participants with limited compute. However, allowing pretrained medical models (which themselves were trained on millions of X-rays) feels contradictory to your previous statement.
Also,since many foundation models (EVA-X, CheXZero, CheXFound, etc.) are trained on huge public datasets, what are the chances that the competition’s test set may already be part of their pretraining data(as stated by @ambrosm )? If that’s possible, wouldn’t it raise concerns about data leakage or unfair advantage?
", "votes": 0, "replies": [ { "id": 3286137, "author_name": "Guntas Dhanjal", "author_url": "", "post_date": "2025-09-09T09:39:05.023000", "content": "Thanks @adarsh1403 for the thoughtful question.
\nWe understand the concern regarding pretrained medical models and potential overlap with unseen test data. The key principle is that all fine-tuning and submissions must be done inside Kaggle, ensuring that every participant works under the same conditions and that results remain reproducible.
\nWhile foundation models may have been trained on large public datasets, participants must comply with the rule that only publicly accessible data and models are used, and no private or competition-specific test data may be incorporated during training. This approach maintains fairness, even when pretrained models are leveraged.
\nAny questions about whether a specific model or dataset is allowed can be asked in the forums to ensure clarity for all participants.
", "votes": -1, "replies": [] }, { "id": 3300861, "author_name": "SonNguyenK17", "author_url": "", "post_date": "2025-10-11T19:40:22.627000", "content": "@guntasdhanjal I would like to know if it is permissible to use Division A data for training a model intended for Division B
", "votes": 0, "replies": [] }, { "id": 3301380, "author_name": "Guntas Dhanjal", "author_url": "", "post_date": "2025-10-13T08:48:55.717000", "content": "Hi @sonnguyenk17 ,
\nEach division has its own independent dataset — the images in Division A and Division B are completely different.
\nTo ensure fairness, models for Division B should be trained only on Division B’s training data.
You can, of course, reuse your workflow or architecture between divisions, but data from one division shouldn’t be used to train models for the other.
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3286069": "I'd like to argue that EVA-X should be banned from the competition: It is contradictory that data from division B must not be used for training division A models whereas EVA-X is trained on a superset of that data.\n\nFurthermore, we can question whether it matters that EVA-X's training data include the test dataset of this competition: The golden rule of data science stipulates that models must be always evaluated on held-out data. Scoring a model on its own training data is meaningless and doesn't say anything about its generalization ability. The authors of EVA-X were aware of the issue and in their own experiments applied the principle: *We do not use any of the images tested subsequently for training, even though they are unlabeled.* Even if a model derived from EVA-X scores high on the private leaderboard, we cannot be sure that it will generalize to unseen data.\n\nFor those who want to learn more about this interesting model: [EVA-X: A foundation model for general chest X-ray analysis with self-supervised learning](https://arxiv.org/pdf/2405.05237)", "3286038": "@guntasdhanjal need some clarification about “no external data”\n\n1.Pretraining on non-medical datasets\nAre models that are pretrained on generic datasets like ImageNet (e.g., EfficientNet, ConvNeXt, ViT, ResNet) allowed? These models are trained on natural images, not medical images.\n\n2.Pretraining on medical datasets\nAre models that are pretrained on medical imaging datasets (e.g., chest X-ray foundation models like EVA-X, CheXZero, or similar) allowed? These have been trained directly on chest X-ray data outside of the competition dataset.\n\nCould you please confirm whether any(or both) of these cases fall under the “external data” restriction?\n", "3286120": "Thanks @adarsh1403 for the great question! Here’s the clarification:\n- Pretraining on generic datasets (like ImageNet) is allowed. Transfer learning from non-medical sources is standard practice and not considered “external data.”\n\n- Pretraining on publicly available medical datasets (e.g., EVA-X, CheXZero, CheXFound, etc.) is also allowed, since these models are openly accessible to all participants. The key requirement is that your final fine-tuning and submission must be done inside Kaggle so results remain fair and reproducible." } }