Hi. This is synthetic data competition.
\nI think most of Kagglers have bitter experience in synthetic data since synthetic data tend to contain artifact or leaks due to processing data.
Leaks causes sometimes catastrophic effects to not only host but participants: e.g. extend of competition term, re-training models, winning solution turned out to be garbage etc.
\nSo I think we must first carefully check whether it has leaks/artifacts before working on this competition task.
\nThis is a list of synthetic data competition I previously participated in which has leaks/artifacts.
\nI hope someone will take hint from that.
Feel free to add ones to the list if you experienced other competitions which includes leaks/artifacts.
\nLastly, I experienced never ending discussion about reporting leaks before competition end.
\nHowever, I personally endorse reporting them since I believe both hosts and participants will take advantage from that in the long run. If participants continuing to hiding leak until competition ends, no host would think they want to hold competition on the Kaggle platform.
Hi. This is synthetic data competition.
\nI think most of Kagglers have bitter experience in synthetic data since synthetic data tend to contain artifact or leaks due to processing data.
Leaks causes sometimes catastrophic effects to not only host but participants: e.g. extend of competition term, re-training models, winning solution turned out to be garbage etc.
\nSo I think we must first carefully check whether it has leaks/artifacts before working on this competition task.
\nThis is a list of synthetic data competition I previously participated in which has leaks/artifacts.
\nI hope someone will take hint from that.
Feel free to add ones to the list if you experienced other competitions which includes leaks/artifacts.
\nLastly, I experienced never ending discussion about reporting leaks before competition end.
\nHowever, I personally endorse reporting them since I believe both hosts and participants will take advantage from that in the long run. If participants continuing to hiding leak until competition ends, no host would think they want to hold competition on the Kaggle platform.
Hi Bilzard, thanks for curating the list and agree with what you said. As organizers, we were aware of the potential for data leaks when we decided to host this competition with a synthetic dataset. While we've made every effort to ensure the competition is leak-proof, nothing is perfect in reality.
\nIn the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden. It helps us ensure our data generation pipeline is robust and as bug-free as possible, and allows us to identify promising and innovative solutions to the problem at hand.
", "rawMarkdown": "Hi Bilzard, thanks for curating the list and agree with what you said. As organizers, we were aware of the potential for data leaks when we decided to host this competition with a synthetic dataset. While we've made every effort to ensure the competition is leak-proof, nothing is perfect in reality. \n\nIn the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden. It helps us ensure our data generation pipeline is robust and as bug-free as possible, and allows us to identify promising and innovative solutions to the problem at hand.\n\n", "votes": 12, "replies": [ { "id": 2945001, "postDate": "2024-08-03T00:21:40.170Z", "content": "\n\n\nIn the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden. It helps us ensure our data generation pipeline is robust and as bug-free as possible, and allows us to identify promising and innovative solutions to the problem at hand.
\n
Thanks. It would definitely help.
\nFor “hiding leaks”, I meant the situation that some teams exploit leaks and have advantage from others and they do not report them in order to keep their benefit. In the long history of Kaggle competition, this kind of strategy is always considered “legal” and they can receive prize without disqualifying. This is because, in my understanding, Kaggle has no rule to ban this action. Therefore, unfortunately, reporting leaks before competition end is rare event. I post this topic hoping this situation never occur.
", "rawMarkdown": "@gordonyip \n\n> In the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden. It helps us ensure our data generation pipeline is robust and as bug-free as possible, and allows us to identify promising and innovative solutions to the problem at hand.\n\nThanks. It would definitely help.\n\nFor “hiding leaks”, I meant the situation that some teams exploit leaks and have advantage from others and they do not report them in order to keep their benefit. In the long history of Kaggle competition, this kind of strategy is always considered “legal” and they can receive prize without disqualifying. This is because, in my understanding, Kaggle has no rule to ban this action. Therefore, unfortunately, reporting leaks before competition end is rare event. I post this topic hoping this situation never occur.", "votes": 2, "replies": [ { "id": 2945625, "postDate": "2024-08-03T15:53:15.507Z", "content": "What if they add a rule where, if the winner does use the leak but reveals their solution and it shows that they took advantage of the leak they would not be awarded the prize?
", "rawMarkdown": "What if they add a rule where, if the winner does use the leak but reveals their solution and it shows that they took advantage of the leak they would not be awarded the prize?", "votes": 1 }, { "id": 2945647, "postDate": "2024-08-03T16:08:58.307Z", "content": "@feidawei
\nI don't know why, but I never heard this kind of rule's been added as long as I know.
@mylesoneill @maggiemd Do you have official comment for that?
\n\n\nWhat if they add a rule where, if the winner does use the leak but reveals their solution and it shows that they took advantage of the leak they would not be awarded the prize?
\n
Edit: c.f. In the competition rule page, it says:
\n\n\n10. NOTIFICATION OF WINNERS & DISQUALIFICATION.
\n
\n Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.
Doesn't exploiting leaks considered to be unfair playing practices or abuses?
", "rawMarkdown": "@feidawei \nI don't know why, but I never heard this kind of rule's been added as long as I know.\n\n@mylesoneill @maggiemd Do you have official comment for that?\n\n> What if they add a rule where, if the winner does use the leak but reveals their solution and it shows that they took advantage of the leak they would not be awarded the prize?\n\nEdit: c.f. In the competition rule page, it says:\n\n> 10\\. NOTIFICATION OF WINNERS & DISQUALIFICATION.\n> Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.\n\nDoesn't exploiting leaks considered to be *unfair playing practices or abuses*?", "votes": 1 }, { "id": 2948098, "postDate": "2024-08-05T16:57:58.533Z", "content": "Hi @tatamikenn, reposting the same comment from another thread, just to keep everyone in the loop:
\n\n", "rawMarkdown": "Hi @tatamikenn, reposting the same comment from another thread, just to keep everyone in the loop:\n\n>thanks for flagging it up to us, we ‘re very happy to be aware early, before we encounter such a problem down the line! We have reached out to Kaggle to see if there is anything we can do moving forward. I am also monitoring the discussion on LMSYS to see if there is any good way forward in case of a leak\n\n>In the mean time, please feel free to suggest your thoughts and propose ideas, it will be great for us to hear what you think, hopefully together we will find a best way forward.\n\n\n", "votes": 4 } ] } ] }, { "id": 2964064, "postDate": "2024-08-19T13:46:44.343Z", "content": "thanks for flagging it up to us, we ‘re very happy to be aware early, before we encounter such a problem down the line! We have reached out to Kaggle to see if there is anything we can do moving forward. I am also monitoring the discussion on LMSYS to see if there is any good way forward in case of a leak
\nIn the mean time, please feel free to suggest your thoughts and propose ideas, it will be great for us to hear what you think, hopefully together we will find a best way forward.
\n
Is there any leakage confirmed? Or it will only be disclosed until the end as a 'magic' ? 🤔
", "rawMarkdown": "Is there any leakage confirmed? Or it will only be disclosed until the end as a 'magic' ? 🤔", "replies": [ { "id": 2964137, "postDate": "2024-08-19T15:01:09.790Z", "content": "No. I’d rather encourage people reporting leakage before competition closed.
", "rawMarkdown": "No. I’d rather encourage people reporting leakage before competition closed.", "votes": 1 }, { "id": 2971609, "postDate": "2024-08-27T13:16:12.870Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 2944972, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-02T22:55:11.893000", "content": "Hi Bilzard, thanks for curating the list and agree with what you said. As organizers, we were aware of the potential for data leaks when we decided to host this competition with a synthetic dataset. While we've made every effort to ensure the competition is leak-proof, nothing is perfect in reality.
\nIn the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden. It helps us ensure our data generation pipeline is robust and as bug-free as possible, and allows us to identify promising and innovative solutions to the problem at hand.
", "votes": 12, "replies": [ { "id": 2945001, "author_name": "Bilzard", "author_url": "", "post_date": "2024-08-03T00:21:40.170000", "content": "\n\n\nIn the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden. It helps us ensure our data generation pipeline is robust and as bug-free as possible, and allows us to identify promising and innovative solutions to the problem at hand.
\n
Thanks. It would definitely help.
\nFor “hiding leaks”, I meant the situation that some teams exploit leaks and have advantage from others and they do not report them in order to keep their benefit. In the long history of Kaggle competition, this kind of strategy is always considered “legal” and they can receive prize without disqualifying. This is because, in my understanding, Kaggle has no rule to ban this action. Therefore, unfortunately, reporting leaks before competition end is rare event. I post this topic hoping this situation never occur.
", "votes": 2, "replies": [ { "id": 2945625, "author_name": "Feida Wei", "author_url": "", "post_date": "2024-08-03T15:53:15.507000", "content": "What if they add a rule where, if the winner does use the leak but reveals their solution and it shows that they took advantage of the leak they would not be awarded the prize?
", "votes": 1, "replies": [] }, { "id": 2945647, "author_name": "Bilzard", "author_url": "", "post_date": "2024-08-03T16:08:58.307000", "content": "@feidawei
\nI don't know why, but I never heard this kind of rule's been added as long as I know.
@mylesoneill @maggiemd Do you have official comment for that?
\n\n\nWhat if they add a rule where, if the winner does use the leak but reveals their solution and it shows that they took advantage of the leak they would not be awarded the prize?
\n
Edit: c.f. In the competition rule page, it says:
\n\n\n10. NOTIFICATION OF WINNERS & DISQUALIFICATION.
\n
\n Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.
Doesn't exploiting leaks considered to be unfair playing practices or abuses?
", "votes": 1, "replies": [] }, { "id": 2948098, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-05T16:57:58.533000", "content": "Hi @tatamikenn, reposting the same comment from another thread, just to keep everyone in the loop:
\n\n", "votes": 4, "replies": [] } ] } ] }, { "id": 2964064, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-08-19T13:46:44.343000", "content": "thanks for flagging it up to us, we ‘re very happy to be aware early, before we encounter such a problem down the line! We have reached out to Kaggle to see if there is anything we can do moving forward. I am also monitoring the discussion on LMSYS to see if there is any good way forward in case of a leak
\nIn the mean time, please feel free to suggest your thoughts and propose ideas, it will be great for us to hear what you think, hopefully together we will find a best way forward.
\n
Is there any leakage confirmed? Or it will only be disclosed until the end as a 'magic' ? 🤔
", "votes": 0, "replies": [ { "id": 2964137, "author_name": "Bilzard", "author_url": "", "post_date": "2024-08-19T15:01:09.790000", "content": "No. I’d rather encourage people reporting leakage before competition closed.
", "votes": 1, "replies": [] }, { "id": 2971609, "author_name": "", "author_url": "", "post_date": "2024-08-27T13:16:12.870000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2944234": "Hi. This is synthetic data competition.\nI think most of Kagglers have bitter experience in synthetic data since synthetic data tend to contain artifact or leaks due to processing data.\n\nLeaks causes sometimes catastrophic effects to not only host but participants: e.g. extend of competition term, re-training models, winning solution turned out to be garbage etc.\n\nSo I think we must first carefully check whether it has leaks/artifacts before working on this competition task.\n\nThis is a list of synthetic data competition I previously participated in which has leaks/artifacts.\nI hope someone will take hint from that.\n\n1. [SETI Breakthrough Listen - E.T. Signal Search](https://www.kaggle.com/competitions/seti-breakthrough-listen)\n - leak1: we can predict labels almost perfectly from the metadata of files (filename, file size, timestamps, histogram of image etc.)\n - leak2: most of background image is duplicated both on train/test data\n2. [Foursquare - Location Matching](https://www.kaggle.com/competitions/foursquare-location-matching)\n - leak: about 66.6% of samples are duplicated on both train/test data\n3. [LEAP - Atmospheric Physics using AI (ClimSim)](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim)\n - leak: location and time data is perfectly reverse engineering from the order of test samples\n\nFeel free to add ones to the list if you experienced other competitions which includes leaks/artifacts.\n\nLastly, I experienced never ending discussion about reporting leaks before competition end.\nHowever, I personally endorse reporting them since I believe both hosts and participants will take advantage from that in the long run. If participants continuing to hiding leak until competition ends, no host would think they want to hold competition on the Kaggle platform.", "2944972": "Hi Bilzard, thanks for curating the list and agree with what you said. As organizers, we were aware of the potential for data leaks when we decided to host this competition with a synthetic dataset. While we've made every effort to ensure the competition is leak-proof, nothing is perfect in reality. \n\nIn the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden. It helps us ensure our data generation pipeline is robust and as bug-free as possible, and allows us to identify promising and innovative solutions to the problem at hand.\n\n", "2964064": "Is there any leakage confirmed? Or it will only be disclosed until the end as a 'magic' ? 🤔" } }