Recently, came across two competitions, LEAP - Atmospheric Physics using AI (ClimSim), where after a data leakage was caught, the competition deadline was extended by the hosts but a few dubious accounts went on to purposefully publish the leaked data a day before the extended deadline. To worse it all no action was taken or checks were performed to see if the leakage was used in the solution which was unfair to those who didn't use it.
\nAnother competition that's concluding tomorrow : LMSYS - Chatbot Arena Human Preference Predictions is meeting the same fate as somebody leaked the data right before the deadline (it's only a few hours from now), participants are waiting for response from the hosts but it's none as of now. All of this takes away from the kagglers who honestly spent their time and resources for months only to get beaten by a leaked solution.
\nThis brings me to open this thread beforehand, in case any of these scenarios repeat, what's expected from the participants? Also, can there be a strict guideline from the hosts so that it doesn't hang on till the very last day only for everybody's efforts to go in vain. I feel this is in good faith for both the kagglers and the hosts who have put in so much efforts to bring this competition to all of us, given it accounts for such a prestigious conference.
", "messages": [ { "id": 2947967, "postDate": "2024-08-05T15:26:30.187Z", "content": "Recently, came across two competitions, LEAP - Atmospheric Physics using AI (ClimSim), where after a data leakage was caught, the competition deadline was extended by the hosts but a few dubious accounts went on to purposefully publish the leaked data a day before the extended deadline. To worse it all no action was taken or checks were performed to see if the leakage was used in the solution which was unfair to those who didn't use it.
\nAnother competition that's concluding tomorrow : LMSYS - Chatbot Arena Human Preference Predictions is meeting the same fate as somebody leaked the data right before the deadline (it's only a few hours from now), participants are waiting for response from the hosts but it's none as of now. All of this takes away from the kagglers who honestly spent their time and resources for months only to get beaten by a leaked solution.
\nThis brings me to open this thread beforehand, in case any of these scenarios repeat, what's expected from the participants? Also, can there be a strict guideline from the hosts so that it doesn't hang on till the very last day only for everybody's efforts to go in vain. I feel this is in good faith for both the kagglers and the hosts who have put in so much efforts to bring this competition to all of us, given it accounts for such a prestigious conference.
", "rawMarkdown": "Recently, came across two competitions, **LEAP - Atmospheric Physics using AI (ClimSim)**, where after a data leakage was caught, the competition deadline was extended by the hosts but a few dubious accounts went on to purposefully publish the leaked data a day before the extended deadline. To worse it all no action was taken or checks were performed to see if the leakage was used in the solution which was unfair to those who didn't use it. \n\nAnother competition that's concluding tomorrow : **LMSYS - Chatbot Arena Human Preference Predictions** is meeting the same fate as somebody leaked the data right before the deadline (it's only a few hours from now), participants are waiting for response from the hosts but it's none as of now. All of this takes away from the kagglers who honestly spent their time and resources for months only to get beaten by a leaked solution. \n\nThis brings me to open this thread beforehand, in case any of these scenarios repeat, what's expected from the participants? Also, can there be a strict guideline from the hosts so that it doesn't hang on till the very last day only for everybody's efforts to go in vain. I feel this is in good faith for both the kagglers and the hosts who have put in so much efforts to bring this competition to all of us, given it accounts for such a prestigious conference. \n\n", "votes": 11 }, { "id": 2948944, "postDate": "2024-08-06T09:15:35.327Z", "content": "Asking the right questionsđŸ”®
", "rawMarkdown": "Asking the right questionsđŸ”®", "votes": 1 }, { "id": 2948026, "postDate": "2024-08-05T16:08:46.110Z", "content": "We had some discussion about it here: https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/523708
", "rawMarkdown": "We had some discussion about it here: https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/523708", "votes": 1, "replies": [ { "id": 2948048, "postDate": "2024-08-05T16:25:35.323Z", "content": "Hey! @feidawei
\nThanks for pointing it out. Just looked into it.
It is said :
\n\n\nIn the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden.
\n
But, the problem is only a very few go ahead with communicating the leaks and as seen in the previous competitions, alot of them incorporate these leaks in their solutions and actually end up scoring medals and wins without exposing them :')
", "rawMarkdown": "Hey! @feidawei \nThanks for pointing it out. Just looked into it. \n\nIt is said :\n>In the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden.\n\nBut, the problem is only a very few go ahead with communicating the leaks and as seen in the previous competitions, alot of them incorporate these leaks in their solutions and actually end up scoring medals and wins without exposing them :')", "votes": 1, "replies": [ { "id": 2948050, "postDate": "2024-08-05T16:27:26.227Z", "content": "Yes, that is still a problem. I tried to ask if we could have additional rules about this. But so far we are just waiting for a response from the hosts.
", "rawMarkdown": "Yes, that is still a problem. I tried to ask if we could have additional rules about this. But so far we are just waiting for a response from the hosts." }, { "id": 2948064, "postDate": "2024-08-05T16:31:40.130Z", "content": "Yeah, a clear guideline from hosts would benefit everybody since the competiton has only started. It would save all of us alot of trouble! :)
\nPeople seemed quite annoyed in previous competitions especially when it happened on the very last day! đŸ¤•
\nHopefully, they'll respond soon!
Hi @feidawei, @ahsuna123 and others
\nthanks for flagging it up to us, we ‘re very happy to be aware early, before we encounter such a problem down the line! We have reached out to Kaggle to see if there is anything we can do moving forward. I am also monitoring the discussion on LMSYS to see if there is any good way forward in case of a leak
\nIn the mean time, please feel free to suggest your thoughts and propose ideas, it will be great for us to hear what you think, hopefully together we will find a best way forward.
", "rawMarkdown": "Hi @feidawei, @ahsuna123 and others\n\nthanks for flagging it up to us, we ‘re very happy to be aware early, before we encounter such a problem down the line! We have reached out to Kaggle to see if there is anything we can do moving forward. I am also monitoring the discussion on LMSYS to see if there is any good way forward in case of a leak\n\nIn the mean time, please feel free to suggest your thoughts and propose ideas, it will be great for us to hear what you think, hopefully together we will find a best way forward.\n\n", "votes": 6 }, { "id": 2948101, "postDate": "2024-08-05T17:02:56.097Z", "content": "Thank you for taking it in consideration! @gordonyip :)
\nFor sure!
Asking the right questionsđŸ”®
", "votes": 1, "replies": [] }, { "id": 2948026, "author_name": "Feida Wei", "author_url": "", "post_date": "2024-08-05T16:08:46.110000", "content": "We had some discussion about it here: https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/523708
", "votes": 1, "replies": [ { "id": 2948048, "author_name": "AC", "author_url": "", "post_date": "2024-08-05T16:25:35.323000", "content": "Hey! @feidawei
\nThanks for pointing it out. Just looked into it.
It is said :
\n\n\nIn the event that a leak does occur (which we hope it is not the case), we welcome exposing them rather than having them hidden.
\n
But, the problem is only a very few go ahead with communicating the leaks and as seen in the previous competitions, alot of them incorporate these leaks in their solutions and actually end up scoring medals and wins without exposing them :')
", "votes": 1, "replies": [ { "id": 2948050, "author_name": "Feida Wei", "author_url": "", "post_date": "2024-08-05T16:27:26.227000", "content": "Yes, that is still a problem. I tried to ask if we could have additional rules about this. But so far we are just waiting for a response from the hosts.
", "votes": 0, "replies": [] }, { "id": 2948064, "author_name": "AC", "author_url": "", "post_date": "2024-08-05T16:31:40.130000", "content": "Yeah, a clear guideline from hosts would benefit everybody since the competiton has only started. It would save all of us alot of trouble! :)
\nPeople seemed quite annoyed in previous competitions especially when it happened on the very last day! đŸ¤•
\nHopefully, they'll respond soon!
Hi @feidawei, @ahsuna123 and others
\nthanks for flagging it up to us, we ‘re very happy to be aware early, before we encounter such a problem down the line! We have reached out to Kaggle to see if there is anything we can do moving forward. I am also monitoring the discussion on LMSYS to see if there is any good way forward in case of a leak
\nIn the mean time, please feel free to suggest your thoughts and propose ideas, it will be great for us to hear what you think, hopefully together we will find a best way forward.
", "votes": 6, "replies": [] }, { "id": 2948101, "author_name": "AC", "author_url": "", "post_date": "2024-08-05T17:02:56.097000", "content": "Thank you for taking it in consideration! @gordonyip :)
\nFor sure!