imposter.png\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F805282%2Fadbf8953c0c85ed7c1d36c94d8128c88%2Fimposter.png?generation=1720748196249711&alt=media\")
It turns out that due to pbuf_ozone_2 leak, test set timestamp and location can be exactly recovered. This leak enables 2D->1D models and possibly also future data utilization (using the input for future samples to 'predict' the output of a previous one).
\nI humbly request the hosts disqualify all techniques that explicitly utilize this (and possibly other similar) leak. It was repeatedly stressed by the hosts that this competition is to be a 1D->1D regression problem, and I, and I think this is also true for many others, invested extensive time and effort (and possibly money for compute) based on this assumption. All of this will become meaningless now if teams start to utilize the leak.
\nOn a personal note, I made a lot of interesting things and utilized advanced techniques in this competition, which I would keep secret for future competitions if I drop from the price range due to a leak, as I'm sure that those techniques would easily lend me gold in future relevant competitions.
It turns out that due to pbuf_ozone_2 leak, test set timestamp and location can be exactly recovered. This leak enables 2D->1D models and possibly also future data utilization (using the input for future samples to 'predict' the output of a previous one).
\nI humbly request the hosts disqualify all techniques that explicitly utilize this (and possibly other similar) leak. It was repeatedly stressed by the hosts that this competition is to be a 1D->1D regression problem, and I, and I think this is also true for many others, invested extensive time and effort (and possibly money for compute) based on this assumption. All of this will become meaningless now if teams start to utilize the leak.
\nOn a personal note, I made a lot of interesting things and utilized advanced techniques in this competition, which I would keep secret for future competitions if I drop from the price range due to a leak, as I'm sure that those techniques would easily lend me gold in future relevant competitions.
From Section B.10 :
\n\"Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.
\nA disqualified participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.\"
\nWe will disqualify leak-based solutions for cash prizes. To be eligible for a cash prize, Kaggle teams will have to verify their training and inference code with the competition hosts.
", "rawMarkdown": "From Section B.10 :\n\n\"Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.\n\nA disqualified participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.\"\n\nWe will disqualify leak-based solutions for cash prizes. To be eligible for a cash prize, Kaggle teams will have to verify their training and inference code with the competition hosts.", "votes": 28, "replies": [ { "id": 2917481, "postDate": "2024-07-11T16:42:11.597Z", "content": "Not to undermine this decision, and honestly it is the best possible outcome for me.
\nI just want to ask, are you sure this task is feasible given the complexities involved?
\nSome teams might use ensembles of tens of models, each requiring heavy computing and long training time, using complex pipelines involving pseudo-labeling, distillation, etc. Easy to hide something behind a lot of compute and complex code. I don't want to lose to some good-at-hiding team; in this case, I would prefer to have an extension and leak allowed and train leaky solution (from scratch, in my case).
\np.s. Surprisingly (or not?) I believe my own solution would be easy to verify, lol.
You could technically build a pipeline to get leak based predictions for the test-set, and only use these predictions to validate models and new ideas during training. I can imagine this could give a small boost but would not be detectable. I think there are still many ways you can gain an unfair advantage which are hard to catch
", "rawMarkdown": "You could technically build a pipeline to get leak based predictions for the test-set, and only use these predictions to validate models and new ideas during training. I can imagine this could give a small boost but would not be detectable. I think there are still many ways you can gain an unfair advantage which are hard to catch", "votes": 1 }, { "id": 2917503, "postDate": "2024-07-11T16:50:12.493Z", "content": "There are many Masters and Grandmasters in top teams. I think they won't give a try to use leak at risk of totally losing their reputation in kaggle.
", "rawMarkdown": "There are many Masters and Grandmasters in top teams. I think they won't give a try to use leak at risk of totally losing their reputation in kaggle.", "votes": 1 }, { "id": 2917525, "postDate": "2024-07-11T16:55:46.543Z", "content": "To alleviate the pressure on the host to conduct checks independently, I propose that all award candidates be required to open-source all their training codes, models, and inference scripts. This way, everyone can inspect each other's work and even conduct random checks.
", "rawMarkdown": "To alleviate the pressure on the host to conduct checks independently, I propose that all award candidates be required to open-source all their training codes, models, and inference scripts. This way, everyone can inspect each other's work and even conduct random checks. " }, { "id": 2917528, "postDate": "2024-07-11T16:57:29.860Z", "content": "Hi @shlomoron , considering that most of the top-ranked teams are Grandmasters or Masters on the verge of becoming Grandmasters, I believe the risk of them hiding leaks is minimal. I trust that the vast majority of Grandmasters, along with the masters hoping to reach gm, care deeply about their reputation within the community. It wouldn't be worth gambling for this single competition with their GM's reputation.
", "rawMarkdown": "Hi @shlomoron , considering that most of the top-ranked teams are Grandmasters or Masters on the verge of becoming Grandmasters, I believe the risk of them hiding leaks is minimal. I trust that the vast majority of Grandmasters, along with the masters hoping to reach gm, care deeply about their reputation within the community. It wouldn't be worth gambling for this single competition with their GM's reputation.", "votes": -1 }, { "id": 2917540, "postDate": "2024-07-11T16:59:55.937Z", "content": "\n\nThere are many Masters and Grandmasters in top teams. I think they won't give a try to use leak at risk of totally losing their reputation in kaggle.
\n
Well, there were cases of kaggle GMs cheating so everything is possible…
", "rawMarkdown": "> There are many Masters and Grandmasters in top teams. I think they won't give a try to use leak at risk of totally losing their reputation in kaggle.\n\nWell, there were cases of [kaggle GMs cheating](https://www.theregister.com/2020/01/21/ai_kaggle_contest_cheat/) so everything is possible...", "votes": 1 }, { "id": 2917542, "postDate": "2024-07-11T17:01:53.677Z", "content": "That's exactly where risk comes from. They might quit kaggle, even lose jobs.
", "rawMarkdown": "That's exactly where risk comes from. They might quit kaggle, even lose jobs.", "votes": 1 }, { "id": 2917544, "postDate": "2024-07-11T17:03:45.093Z", "content": "\n\nTo alleviate the pressure on the host to conduct checks independently, I propose that all award candidates be required to open-source all their training codes, models, and inference scripts. This way, everyone can inspect each other's work and even conduct random checks.
\n
On the contrary, the top 10 mustn't publish their solutions in the first days.
\nImagine this scenario: Team X uses cheating, reads Team Y's legit solution on the forum, and then incorporates Team Y's ideas into his 'solution' that he sends to the host to verify. With new ideas, his score becomes legit.
\nOf course, after host verified it would be good to have a second verification on the public forum.
Your consideration is very thorough. However, we are essentially dealing with a game theory scenario. If the host decides to conduct a thorough review, the likelihood of individuals taking risks will significantly decrease. Considering this, I believe that it would be more fair and economically approach?
", "rawMarkdown": "Your consideration is very thorough. However, we are essentially dealing with a game theory scenario. If the host decides to conduct a thorough review, the likelihood of individuals taking risks will significantly decrease. Considering this, I believe that it would be more fair and economically approach?\n\n\n\n\n", "votes": 2 } ] }, { "id": 2917504, "postDate": "2024-07-11T16:50:13.643Z", "content": "Under which of those criterion are competitors going to be disqualified? Using the data at hand is not cheating.
\nA competition is about taking a pile of data, finding some signal, and generating value. The rules were such that the provided dataset (and more!) were encouraged to be used to make the best possible predictions. While there was indication that the organizers did not want certain types of solutions to be used, those were never formally specified. Multiple questions for clarifications in the forums went ignored. When the dataset was updated, that was the time to make a formal rule change, if required.
\nKaggle: use data, find signal, but thread the needle on which signals are acceptable to use without official codified guidance.
", "rawMarkdown": "Under which of those criterion are competitors going to be disqualified? Using the data at hand is not cheating.\n\nA competition is about taking a pile of data, finding some signal, and generating value. The rules were such that the provided dataset (and more!) were encouraged to be used to make the best possible predictions. While there was indication that the organizers did not want certain types of solutions to be used, those were never formally specified. Multiple questions for clarifications in the forums went ignored. When the dataset was updated, that was the time to make a formal rule change, if required.\n\nKaggle: use data, find signal, but thread the needle on which signals are acceptable to use without official codified guidance. ", "votes": -11, "replies": [ { "id": 2917519, "postDate": "2024-07-11T16:53:36.700Z", "content": "Hey, Please use your main account to talk, and do not use a secondary account to ask questions.
", "rawMarkdown": "Hey, Please use your main account to talk, and do not use a secondary account to ask questions.", "votes": 15 }, { "id": 2917631, "postDate": "2024-07-11T17:45:55.870Z", "content": "This is my only account.
", "rawMarkdown": "This is my only account.", "votes": -10 }, { "id": 2918040, "postDate": "2024-07-12T01:27:42.740Z", "content": "Ok, I notice you have really deep undetstanding about this dataset, so why not give some submits and hava fun?
\nAlso, could you please explain why you leaked leakage in last 5 days and what is your intention?
\nDude, your account behaviou is really strange and I can't help but suspect your are Oliver Queen,the Green Arrow, possessing two faces.
\nimposter.png
\n
Looks like everyone is running out of patience.
\nI think there will be at least several months to verify top teams' work one by one.
\nif this's final decision, i suggest top 5 teams to public their training and inference code, otherwise it's really terrible.
Copying from my other comment: ' top 10 mustn't publish their solutions in the first days.
\nImagine this scenario: Team X uses cheating, reads Team Y's legit solution on the forum, and then incorporates Team Y's ideas into his 'solution' that he sends to the host to verify. With new ideas, his score becomes legit.
\nOf course, after host verified it would be good to have a second verification on the public forum.'
It makes sense!
", "rawMarkdown": "It makes sense!", "votes": 3 }, { "id": 2918165, "postDate": "2024-07-12T04:03:35.317Z", "content": "Smart choice is to avoid publishing solution until the end of code verification.
", "rawMarkdown": "Smart choice is to avoid publishing solution until the end of code verification." } ] }, { "id": 2917692, "postDate": "2024-07-11T18:19:24.557Z", "content": "\nThank you for clarifying the measures to be taken when leaks are used. However, the criteria for what falls under a leak are still unclear, and I would like this to be clarified.
\nFrom your intentions, we understand that using explicit leaks present in the test data (such as pbuf_ozone_2) to predict a single row from multiple rows is prohibited.
\nHowever, for example:
cam_in_LANDFRAC alone is sufficient to reveal the identity of 163 locations. Is anything which gives spatial temporal information allowed?
I understand that the provided features, such as cam_in_LANDFRAC, are sufficient to give the model information about the location. Therefore, I will not include such features in future training. However, what if such features were included in early experiments and left as they were? In this case, there is not enough time to retrain the model according to the rules.
\nAs a solution, those with and without these features are essentially the same, aligning with the host's intentions. However, if this is considered a violation under the newly established rules, will those teams be banned? What I want to clarify is the rules.
On the one hand, this has opened up a new way of dealing with leaks; in the future, competitions with leaks may follow this example, which is the most worrying aspect. On the other hand, the time spent cooperating with the organizers to fully restore the code may not be less than the time we spend re-running the model. Sometimes, due to different environments and configurations, it might not run at all.
", "rawMarkdown": "On the one hand, this has opened up a new way of dealing with leaks; in the future, competitions with leaks may follow this example, which is the most worrying aspect. On the other hand, the time spent cooperating with the organizers to fully restore the code may not be less than the time we spend re-running the model. Sometimes, due to different environments and configurations, it might not run at all.", "votes": 3 } ] }, { "id": 2916872, "postDate": "2024-07-11T10:19:50.783Z", "content": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it. Maybe i am too romantic or too new into this …
", "rawMarkdown": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it. Maybe i am too romantic or too new into this ...", "votes": 9 }, { "id": 2916314, "postDate": "2024-07-10T22:31:07.513Z", "content": "Hello all. I am waiting to confirm this with the Kaggle staff, but it may be safe to assume for now that we will disqualify any submissions that take advantage of any leak or use a multicolumn approach for cash prizes and gold medals.
", "rawMarkdown": "Hello all. I am waiting to confirm this with the Kaggle staff, but it may be safe to assume for now that we will disqualify any submissions that take advantage of any leak or use a multicolumn approach for cash prizes and gold medals.", "votes": 12, "replies": [ { "id": 2916321, "postDate": "2024-07-10T22:38:01.803Z", "content": "Thank you for the answer! Let me clarify. Will it be applied only for the test time? As discussed in other threads, some training techniques unintentionally use other sample information. I believe disqualifying such a solution is not what you intend, but clarification would be better.
", "rawMarkdown": "Thank you for the answer! Let me clarify. Will it be applied only for the test time? As discussed in other threads, some training techniques unintentionally use other sample information. I believe disqualifying such a solution is not what you intend, but clarification would be better.", "votes": 2, "replies": [ { "id": 2916323, "postDate": "2024-07-10T22:45:49.803Z", "content": "Can you explain what you mean by unintentionally use other sample information via an explicit example?
", "rawMarkdown": "Can you explain what you mean by unintentionally use other sample information via an explicit example?", "votes": 1 }, { "id": 2916328, "postDate": "2024-07-10T23:01:01.620Z", "content": "Typical one is batch normalization. Other example is pseudo labeling.
\nDisucussed in the thread.
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519233
And how about explicit use of multi columns information at training time? Like pretraining multi columns model, then fine tuning for single columns setting. Or using nearest columns target as auxiliary loss and so on… (I don't use these though.)
", "rawMarkdown": "Typical one is batch normalization. Other example is pseudo labeling.\nDisucussed in the thread.\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519233\n\nAnd how about explicit use of multi columns information at training time? Like pretraining multi columns model, then fine tuning for single columns setting. Or using nearest columns target as auxiliary loss and so on... (I don't use these though.)", "votes": 1 }, { "id": 2916334, "postDate": "2024-07-10T23:16:36.113Z", "content": "Typically by using other sample information, I am referring to using multiple samples corresponding to the same timestep. It would be difficult to do this unintentionally.
\nWe will likely seek to disqualify submissions that reverse-engineer location / timestep information at any point.
", "rawMarkdown": "Typically by using other sample information, I am referring to using multiple samples corresponding to the same timestep. It would be difficult to do this unintentionally.\n\nWe will likely seek to disqualify submissions that reverse-engineer location / timestep information at any point.", "votes": 3 }, { "id": 2916338, "postDate": "2024-07-10T23:30:12.687Z", "content": "\n\nTypically by using other sample information, I am referring to using multiple samples corresponding to the same timestep. It would be difficult to do this unintentionally.
\n
@jerrylin96
\nTypical machine learning model easy to learn location and timestamp information from pbuf_ozon_2 or other targets it may encode them. That means all participants model more or less utilizes this “leak” if participants know or not. It’s no exception like the baseline model the host prepared. How do you think of that fact?
\n\nTypical machine learning model easy to learn location and timestamp information from pbuf_ozon_2 or other targets it may encode them.
\n
I totally agree.
\nEven if utilization of the specific information like pbuf_ozon_2 will be explicitly prohibited, ML models can learn the timestamp / location info somewhat, because we know how to create training dataset.
I suggest a criteria like below,
\nThis competition is a CSV submission competition, it is quite challenging to investigate whether there has been any leaking solution. Additionally, since the resource consumption of this competition lies in training rather than inference, I suggest the following:
\nThe suggestion proposed by @hengzheng is reasonable. @jerrylin96
", "rawMarkdown": "The suggestion proposed by @hengzheng is reasonable. @jerrylin96 ", "votes": 7 }, { "id": 2916596, "postDate": "2024-07-11T06:01:28.023Z", "content": "IMHO, Coding check need to cover train part. Only checking inference inputs is not enough.
\nOtherwise, kagglers can use 2d -> 1d models to get oof and then use pseudo labels or distillation to learn this oof.
", "rawMarkdown": "IMHO, Coding check need to cover train part. Only checking inference inputs is not enough.\n\nOtherwise, kagglers can use 2d -> 1d models to get oof and then use pseudo labels or distillation to learn this oof.", "votes": 1 } ] }, { "id": 2916394, "postDate": "2024-07-11T01:24:52.700Z", "content": "I declare that I will use the leak unless the competition host makes the rules clear to prohibits the leak. (how about pseudo-labeling? stacking? distilation? batch normalization? any other trick? etc). In a competition, there is no choice but to not spend the time to do EVERYTHING that are necessary to win. If the competition host changes the rules, I will follow the rules, but the new rules should be clear and feasible (to hold fairness.)
\nThe recently concluded Image Matching Challenge 2024 was also won by solutions that were not intended by the host. In some test cases, the problem was solved by estimating the camera order without directly solving the camera pose estimation that should have been solved. There were probably many other competitions where solutions were used that the host did not intend or want.
\nAnyway, I am tired. I want to be free from this competition as soon as possible.
\nActually, we spent not a short period of time (we are concentrating on one or two weeks. we have already found leak independently of yesterday's leak revelations from someone.) building the model utilizing timestamp information because it is not prohibited by rules and any rule change that prohibits certain methods is unfair, at least to us.
", "rawMarkdown": "I declare that I will use the leak unless the competition host makes the rules clear to prohibits the leak. (how about pseudo-labeling? stacking? distilation? batch normalization? any other trick? etc). In a competition, there is no choice but to not spend the time to do EVERYTHING that are necessary to win. If the competition host changes the rules, I will follow the rules, but the new rules should be clear and feasible (to hold fairness.)\n\nThe recently concluded Image Matching Challenge 2024 was also won by solutions that were not intended by the host. In some test cases, the problem was solved by estimating the camera order without directly solving the camera pose estimation that should have been solved. There were probably many other competitions where solutions were used that the host did not intend or want.\n\nAnyway, I am tired. I want to be free from this competition as soon as possible.\n\nActually, we spent not a short period of time (we are concentrating on one or two weeks. we have already found leak independently of yesterday's leak revelations from someone.) building the model utilizing timestamp information because it is not prohibited by rules and any rule change that prohibits certain methods is unfair, at least to us.", "votes": 3, "replies": [ { "id": 2916399, "postDate": "2024-07-11T01:45:27.337Z", "content": "Foursquare again?
", "rawMarkdown": "Foursquare again?", "votes": 4 }, { "id": 2916401, "postDate": "2024-07-11T01:51:28.027Z", "content": "The situations are similar but different. My stomach pain is the same.
", "rawMarkdown": "The situations are similar but different. My stomach pain is the same.", "votes": -1 }, { "id": 2916412, "postDate": "2024-07-11T02:24:20.863Z", "content": "I completely agree with @charmq 's opinion.
\nIf the rules are clarified, I will follow them as well. However, there are no rules prohibiting the use of location or time, including the multi-column to single-column approach. Given that it is difficult to clearly distinguish between prohibited and non-prohibited methods, adding rules that everyone can understand is unrealistic. If these methods are to be prohibited, the competition should be designed from the outset to make their use impossible, such as by using time series API.
Furthermore, if prohibitions are introduced, the ambiguity of the rules could lead to different interpretations, potentially undermining fairness among competitors. We are simply doing our best within the bounds of the current rules, and it is unfair to deny us for that.
", "rawMarkdown": "I completely agree with @charmq 's opinion.\nIf the rules are clarified, I will follow them as well. However, there are no rules prohibiting the use of location or time, including the multi-column to single-column approach. Given that it is difficult to clearly distinguish between prohibited and non-prohibited methods, adding rules that everyone can understand is unrealistic. If these methods are to be prohibited, the competition should be designed from the outset to make their use impossible, such as by using time series API.\n\nFurthermore, if prohibitions are introduced, the ambiguity of the rules could lead to different interpretations, potentially undermining fairness among competitors. We are simply doing our best within the bounds of the current rules, and it is unfair to deny us for that.", "votes": -2 }, { "id": 2916414, "postDate": "2024-07-11T02:27:33.223Z", "content": "If a leak is discovered for the first time, I can understand. But it is the second time, so I cannot understand at all.
\nThe host has emphasized many times, not to use 2d -> 1d solution and the host even changed dataset for this reason. I have no idea why you guys still keep digging this, as you said \"not a short period of time\" and \"we have already found leak independently of yesterday's leak revelations from someone.\".
\nIf you said you used leak because of yesterday's leak revelations from someone, I totally understand
\nWhat confused me is that before yesterday's leak revelations from someone and after the host changed dataset, you still spend 2 weeks to dig leak
\nI don't think you are tired. Instead, I think you are full of energy, you have one or two weeks to dig something forbidden.
\nREALLY confused. :(
", "rawMarkdown": "If a leak is discovered for the first time, I can understand. But it is the second time, so I cannot understand at all.\n\nThe host has emphasized many times, not to use 2d -> 1d solution and the host even changed dataset for this reason. I have no idea why you guys still keep digging this, as you said \"not a short period of time\" and \"we have already found leak independently of yesterday's leak revelations from someone.\". \n\nIf you said you used leak because of yesterday's leak revelations from someone, I totally understand\n\nWhat confused me is that before yesterday's leak revelations from someone and after the host changed dataset, you still spend 2 weeks to dig leak\n\nI don't think you are tired. Instead, I think you are **full of energy**, you have one or two weeks to dig something forbidden. \n\nREALLY confused. :(\n\n\n\n", "votes": 22 }, { "id": 2916417, "postDate": "2024-07-11T02:34:33.443Z", "content": "It's very unfortunate and regrettable, but the host's intention and the competition rules are different things.
\nIf someone does their best within the rules, we also need to do our best to compete.
I think same thing as @kami634 's post. (https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519249#2916412)
\n\n\nGiven that it is difficult to clearly distinguish between prohibited and non-prohibited methods, adding rules that everyone can understand is unrealistic.
\n
It's true that making new rules itself is too difficult and unrealistic.
\n\n\nWe are simply doing our best within the bounds of the current rules, and it is unfair to deny us for that.
\n
If the rules are changed, we are forced to spend time to modify the model to comply with the changes. And the host's acceptance of the model itself is not known until the end (because the criteria are vague).
", "rawMarkdown": "I think same thing as @kami634 's post. (https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519249#2916412)\n>Given that it is difficult to clearly distinguish between prohibited and non-prohibited methods, adding rules that everyone can understand is unrealistic.\n\nIt's true that making new rules itself is too difficult and unrealistic.\n\n>We are simply doing our best within the bounds of the current rules, and it is unfair to deny us for that.\n\nIf the rules are changed, we are forced to spend time to modify the model to comply with the changes. And the host's acceptance of the model itself is not known until the end (because the criteria are vague)." }, { "id": 2916426, "postDate": "2024-07-11T02:44:10.560Z", "content": "A project will last 3 months
\n------------------- The early stage of the project ---------------
\nSomeone1: \"Can we do that? \"
\nBoss: \"No, we don't want that. \"
------------------- The middle stage of the project ---------------
\nSomeone2: \"We have successfully do that.\"
\nBoss: \"No, we don't want that. Since there is no much time left, let's postpone this project for 2 weeks\"
------------------- The last 5 days of the project ---------------
\nSomeone3: \"The rule of this project doesn't forbid this. So we still spend much time on it. Although you don't want that, you have to give us high kpi(high rank) and pay us money. \"
\nBoss:….
Hosts pay money for a solution useful for them. In this case, they pay money for human progress. Since they don't want the solution (And they did repeatedly stress it, again and again and again), I think they at least have the right to refuse to pay the money.
", "rawMarkdown": "A project will last 3 months\n\n------------------- The early stage of the project ---------------\nSomeone1: \"Can we do that? \"\nBoss: \"No, we don't want that. \"\n\n------------------- The middle stage of the project ---------------\nSomeone2: \"We have successfully do that.\"\nBoss: \"No, we don't want that. Since there is no much time left, let's postpone this project for 2 weeks\"\n\n------------------- The last 5 days of the project ---------------\nSomeone3: \"The rule of this project doesn't forbid this. So we still spend much time on it. Although you don't want that, you have to give us high kpi(high rank) and pay us money. \"\nBoss:....\n\nHosts pay money for a solution useful for them. In this case, they pay money for human progress. Since they don't want the solution (And they did repeatedly stress it, **again and again and again**), I think they at least have the right to refuse to pay the money. ", "votes": 10 }, { "id": 2916427, "postDate": "2024-07-11T02:45:55.183Z", "content": "\n\nhost's intention and the competition rules are different things
\n
I thought this is true. If the host explicitly forbidden it earlier in the rules, we did work on another approach.
", "rawMarkdown": "> host's intention and the competition rules are different things\n\nI thought this is true. If the host explicitly forbidden it earlier in the rules, we did work on another approach.", "votes": -2 }, { "id": 2916437, "postDate": "2024-07-11T03:04:38.420Z", "content": "But they did repeatedly say that they don't want that. And they had postponed this competition for this issues 2 weeks. Still, you think it should be allowed.
", "rawMarkdown": "But they did repeatedly say that they don't want that. And they had postponed this competition for this issues 2 weeks. Still, you think it should be allowed.", "votes": 5 }, { "id": 2916440, "postDate": "2024-07-11T03:08:48.717Z", "content": "\n\nthey did repeatedly say that they don't want that
\n
However, the host did not answer anything about the clear prohibition in the rules.
\nTherefore, the interpretation is divided. If we could have clearly heard that it was prohibited, we would have explored a different approach.
\nIf it were to be prohibited, it would not have been necessary to replace the test set…
They can't check all of the messages on the forum. But they have replaced the test set and postpone it for 2 weeks, isn't it clear enough?
", "rawMarkdown": "They can't check all of the messages on the forum. But they have replaced the test set and postpone it for 2 weeks, isn't it clear enough?\n", "votes": 2 }, { "id": 2916443, "postDate": "2024-07-11T03:14:36.647Z", "content": "So if there hadn't been yesterday's reveal post, were you guys planning to hide until the last day or even the last minute, like in the Foursquare?
", "rawMarkdown": "So if there hadn't been yesterday's reveal post, were you guys planning to hide until the last day or even the last minute, like in the Foursquare?", "votes": 8 }, { "id": 2916444, "postDate": "2024-07-11T03:20:30.870Z", "content": "The original top two teams in Deepfake Challenge didn't clearly violate the rules, but they were nonetheless demoted or disqualified in the end. In fact, I think what happened here is far worse than anything in the Deepfakes. So I won't be surprise if the host decide to disqualified the prize and even rank.
", "rawMarkdown": "The original top two teams in Deepfake Challenge didn't clearly violate the rules, but they were nonetheless demoted or disqualified in the end. In fact, I think what happened here is far worse than anything in the Deepfakes. So I won't be surprise if the host decide to disqualified the prize and even rank." }, { "id": 2916445, "postDate": "2024-07-11T03:20:31.887Z", "content": "I totally agree with host's intention and the competition rules are different things.
If you said you used leak because of yesterday's leak revelations from someone, I totally understand
\nWhat confused me is that before yesterday's leak revelations from someone and after the host changed dataset, you still spend 2 weeks to dig leak.
\nActually, I have explored the leak before dataset change. My method is similar but not exactly same with leak revelations from someone. I use t-sne to plot the 60 pbuf_ozone features group. A year data make a circle. Point order is exactly time order. But since host said not to use time order, I didn't spend time digging this after dataset change. It's meaningless.
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5670224%2F7086c0ddb2ae8641f4e0a051a1506b0f%2F7A0BE72C-7CC9-403F-8299-5D476CDD648D.png?generation=1720667650428188&alt=media\")
@hookman
\nYou found the potential of the leak but did not use it. I respect that attitude of yours. (I also learned a lot from your use of t-SNE)
\nI understand host's intenttion. On the other hand, kaggle is a competition. There were several questions in discussion, and I also asked questions directly via email, but they were ignored.
\nAs long as the host did not explicitly state that it is forbidden, we have to consider hacking the leak in order to win the competition.
you were proud of using leak and you just think you fellow rules。Have you no shame?
", "rawMarkdown": "you were proud of using leak and you just think you fellow rules。Have you no shame?", "votes": -8 }, { "id": 2916497, "postDate": "2024-07-11T04:19:07.820Z", "content": "\nTo some extent, I agree with you. Although I doubt intention of digging leak, everyone has different thoughts and it's exactly grey area. I can not judge too much.
\nIt's true that the host did not explicitly state that it is forbidden. But I think even if the host said it's forbidden, it's though very hard to check this. The training is quiet time-consuming and host obviously cannot check everyone's code.
\nLet's just wait and see how host deal with this.
", "rawMarkdown": "@kurupical \n\nTo some extent, I agree with you. Although I doubt intention of digging leak, everyone has different thoughts and it's exactly grey area. I can not judge too much. \n\nIt's true that the host did not explicitly state that it is forbidden. But I think even if the host said it's forbidden, it's though very hard to check this. The training is quiet time-consuming and host obviously cannot check everyone's code.\n\nLet's just wait and see how host deal with this.", "votes": 5 }, { "id": 2916695, "postDate": "2024-07-11T07:25:52.230Z", "content": "I think I would have agreed with this take - but the host just said in the post that you replied to that it wouldn't be permissible..
", "rawMarkdown": "I think I would have agreed with this take - but the host just said in the post that you replied to that it wouldn't be permissible.." }, { "id": 2916862, "postDate": "2024-07-11T10:14:45.607Z", "content": "\n\nIf a leak is discovered for the first time, I can understand. But it is the second time, so I cannot understand at all.
\n
This being the second leak only proves the host's negligence. These leaks should be avoided by the competition design and not by additional rules that can only be enforced arbitrarily. Careful design could have prevented this from the beginning and since they weren't careful enough the first time, the first leak should have been a wakeup call. It's not unreasonable to expect that hosts check that no features map to any parameters they are trying to keep hidden; it should be a mandatory step in setting up any competition really. We've had a string of competitions recently sabotaged by kaggle and the hosts not making the simplest tests and leaving competitions vulnerable.
", "rawMarkdown": ">If a leak is discovered for the first time, I can understand. But it is the second time, so I cannot understand at all.\n\nThis being the second leak only proves the host's negligence. These leaks should be avoided by the competition design and not by additional rules that can only be enforced arbitrarily. Careful design could have prevented this from the beginning and since they weren't careful enough the first time, the first leak should have been a wakeup call. It's not unreasonable to expect that hosts check that no features map to any parameters they are trying to keep hidden; it should be a mandatory step in setting up any competition really. We've had a string of competitions recently sabotaged by kaggle and the hosts not making the simplest tests and leaving competitions vulnerable. " } ] }, { "id": 2916501, "postDate": "2024-07-11T04:21:42.060Z", "content": "Will the competition be ended soon, or will keep the ddl unchanged?
", "rawMarkdown": "Will the competition be ended soon, or will keep the ddl unchanged?", "replies": [ { "id": 2916540, "postDate": "2024-07-11T04:50:09.317Z", "content": "It's impossible to end immediately. But disqualified the rank and money is possible.
", "rawMarkdown": "It's impossible to end immediately. But disqualified the rank and money is possible." } ] } ] }, { "id": 2915289, "postDate": "2024-07-10T12:32:56.197Z", "content": "New Competition Idea: Predict the next deadline extension
", "rawMarkdown": "**New Competition Idea: Predict the next deadline extension**", "votes": 8, "replies": [ { "id": 2915323, "postDate": "2024-07-10T12:50:52.013Z", "content": "We are sooo tired tho 😭
", "rawMarkdown": "We are sooo tired tho 😭", "votes": 4 } ] }, { "id": 2916993, "postDate": "2024-07-11T12:09:23.783Z", "content": "@jerrylin96
\nWhatever you decide, I hope you will not extend the competition. You previously referred to the 2 week extension as \"good news\", but it is clearly \"bad news\" as evidenced by participents reactions.
\nWe kagglers are more committed to kaggle than you might think. We are working as hard as we can towards the end of the competition period. We have a schedule that is tailored to the competition.
\nIn the past, when the competition was extended, some people said \"I was planning a trip right after the competition was over\"(*1)
Extending this further would be like running a full marathon, reaching the 40 km mark, and starting the final sprint, only to announce that the finish line has been changed to 50 km.
\n(*1) https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/110777#637593
", "rawMarkdown": "@jerrylin96 \nWhatever you decide, I hope you will not extend the competition. You previously referred to the 2 week extension as \"good news\", but it is clearly \"bad news\" as evidenced by participents reactions. \nWe kagglers are more committed to kaggle than you might think. We are working as hard as we can towards the end of the competition period. We have a schedule that is tailored to the competition. \nIn the past, when the competition was extended, some people said \"I was planning a trip right after the competition was over\"(*1) \n\nExtending this further would be like running a full marathon, reaching the 40 km mark, and starting the final sprint, only to announce that the finish line has been changed to 50 km.\n\n(*1) https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/110777#637593", "votes": 6, "replies": [ { "id": 2917001, "postDate": "2024-07-11T12:19:05.393Z", "content": "I would have to disagree here. It's better to have an extended good competition than a finished-in-time bad one. We will just need to stay strong and persevere.
", "rawMarkdown": "I would have to disagree here. It's better to have an extended good competition than a finished-in-time bad one. We will just need to stay strong and persevere. ", "votes": 8, "replies": [ { "id": 2917034, "postDate": "2024-07-11T12:38:10.987Z", "content": "\n\nelse: nothing has changed in the situation, so no extension is needed regardless of whether the host allows the use of leaks or not
\n
A LOT has changed. If leak is allowed, it's in the best interest of both most competitors (who only now know of leak) and the host (who want to get the best possible solutions, whether based on leak or not) to extend. The ONLY people with whom an extension would not be in their interest is the tiny minority (your team included, mind you) that already have a leak-based solution. It's a VERY different situation from what you referred to in fraud detection. And I believe the community would agree with me on this.
", "rawMarkdown": ">else: nothing has changed in the situation, so no extension is needed regardless of whether the host allows the use of leaks or not\n\nA LOT has changed. If leak is allowed, it's in the best interest of both most competitors (who only now know of leak) and the host (who want to get the best possible solutions, whether based on leak or not) to extend. The ONLY people with whom an extension would not be in their interest is the tiny minority (your team included, mind you) that already have a leak-based solution. It's a VERY different situation from what you referred to in fraud detection. And I believe the community would agree with me on this.", "votes": 3 }, { "id": 2917052, "postDate": "2024-07-11T13:01:41.020Z", "content": "I consider the current situation to be \"unwanted but not prohibited\" and in that sense I understand that the situation has not changed. (but this is my opinion and I'm sure there are many different arguments) From your past statements, I mistakenly thought that you had the same opinion as mine. If you consider the current situiation to be \"unwanted and prohibited\" and think from your point of view, your statement is right.
\nKaggle very simple, really. EVERYTHING allowed… except what written Rules .\nDoes anything said , 'multi-row--single-target approach forbidden'?\nNo. Hence, allowed.\nyes but the host declared clearly that multi-to-single are not allowed. This might not be writen but it is obivioustly a new rule, no?
", "rawMarkdown": "yes but the host declared clearly that multi-to-single are not allowed. This might not be writen but it is obivioustly a new rule, no?" }, { "id": 2917067, "postDate": "2024-07-11T13:08:10.607Z", "content": "I agree with you that it is \"unwanted but not prohibited\". Since it is unwanted, I prefer to change the situation to be 'wanted'. Using leak is OK but it's a risk if leak is leaked before the end of the competition. This is also part of the game IMO.
", "rawMarkdown": "I agree with you that it is \"unwanted but not prohibited\". Since it is unwanted, I prefer to change the situation to be 'wanted'. Using leak is OK but it's a risk if leak is leaked before the end of the competition. This is also part of the game IMO.", "votes": 1 }, { "id": 2917075, "postDate": "2024-07-11T13:17:55.463Z", "content": "@vasileioscharatsidis Yes, I agree with you.
", "rawMarkdown": "@vasileioscharatsidis Yes, I agree with you. ", "votes": 1 }, { "id": 2917080, "postDate": "2024-07-11T13:23:42.517Z", "content": "\n\nSince it is unwanted, I prefer to change the situation to be 'wanted'.
\n
I understand 'wanted' -> 'host wanted'.
\nIf I was a host and be forced to allowed to use leak, I don't want to extent the deadline. Because whether extenting or not, there will be only solution using leak. (But 'some participants wanted' to extent in allowed case.)
\n\nUsing leak is OK but it's a risk if leak is leaked before the end of the competition. This is also part of the game IMO.
\n
I agree!
", "rawMarkdown": "> Since it is unwanted, I prefer to change the situation to be 'wanted'. \n\nI understand 'wanted' -> 'host wanted'. \nIf I was a host and be forced to allowed to use leak, I don't want to extent the deadline. Because whether extenting or not, there will be only solution using leak. (But 'some participants wanted' to extent in allowed case.)\n\n> Using leak is OK but it's a risk if leak is leaked before the end of the competition. This is also part of the game IMO. \n\nI agree!", "votes": -2 }, { "id": 2917091, "postDate": "2024-07-11T13:28:31.827Z", "content": "\n\nIf I was a host and be forced to allowed to use leak, I don't want to extent the deadline. Because whether extenting or not, there will be only solution using leak. (But 'some participants wanted' to extent in allowed case.)
\n
Ultimately what host want is the most important.
\nIf host want to finish in time, so be it.
\nBut since you shared why YOU don't want an extension, I will share the other side too.
@jerrylin96 If there exists even one participant who intends to stick with not using the leak and has the balls to try to be competitive even against leak-based approaches, then I feel we should give an extension, even if just for a week.
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519249#2915873
", "rawMarkdown": "@jerrylin96 If there exists even one participant who intends to stick with *not* using the leak and has the balls to try to be competitive even against leak-based approaches, then I feel we should give an extension, even if just for a week.\n\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519249#2915873" }, { "id": 2917107, "postDate": "2024-07-11T13:34:16.850Z", "content": "Of course, if leaks are allowed and there are no extensions, that would be great for me.
\nBut I'm not making this statement because I want to win, I'm just really tired and want it to be over quickly😂
\n\nIf there exists even one participant who intends to stick with not using the leak and has the balls to try to be competitive even against leak-based approaches
\n
Look at the difference between me and current 1st place. And believe me, I did a LOT.
\nThis is how 'competitive' you can be against leak in the high ranks.
\nHowever, extension is good. Since then, I can try to bring a leaky solution too :)
Thank you for sharing your thought.
\nI thought you use leakage until yesterday, so I'm very surprised to know your achievement is without leakage.
Whatever decision the host makes, let's do our best (of course, within the rules!).
\nI'm looking forward to read your solution.
@shlomoron
\nYou are so energetic🤣 To be honest, I have no power or interest in utilizing leaked information. I will give up if the host decides to allow leaky solutions. (But I am betting that the host will only disqualify explicit multi-to-single column predictions at test time.)
\n\nI thought you use leakage until yesterday
\n
So you guys tried to use leak because you can't catch up?
", "rawMarkdown": ">I thought you use leakage until yesterday\n\nSo you guys tried to use leak because you can't catch up?", "votes": 1 }, { "id": 2917138, "postDate": "2024-07-11T13:58:21.190Z", "content": "\n\nI thought you use leakage until yesterday, so I'm very surprised to know your achievement is without leakage.
\n
Why you thought so? I'm only ~0.001 above your solution. It's within reason. Yep, my solution is leak-free and is truly a great one if I say so about myself lol. (Also, it will stay a secret if I don't finish in price range 🤣) Anyway, thank you.
", "rawMarkdown": ">I thought you use leakage until yesterday, so I'm very surprised to know your achievement is without leakage.\n\nWhy you thought so? I'm only ~0.001 above your solution. It's within reason. Yep, my solution is leak-free and is truly a great one if I say so about myself lol. (Also, it will stay a secret if I don't finish in price range 🤣) Anyway, thank you.", "votes": 5 }, { "id": 2917143, "postDate": "2024-07-11T14:02:47.983Z", "content": "\n\nYou are so energetic🤣 To be honest, I have no power or interest in utilizing leaked information. I will give up if the host decides to allow leaky solutions
\n
Hahaha I had several great insights and I'm pretty sure they can be carried on at least partly to leaky solutions. Even If I don't succeed to develop one myself, I only need to find a team with working leaky solution and I would probably be able to contribute much. So yeah I might be tired, but I won't go down without a fight lol
", "rawMarkdown": "> You are so energetic🤣 To be honest, I have no power or interest in utilizing leaked information. I will give up if the host decides to allow leaky solutions\n\nHahaha I had several great insights and I'm pretty sure they can be carried on at least partly to leaky solutions. Even If I don't succeed to develop one myself, I only need to find a team with working leaky solution and I would probably be able to contribute much. So yeah I might be tired, but I won't go down without a fight lol" }, { "id": 2917151, "postDate": "2024-07-11T14:07:18.197Z", "content": "This is the third time I read a comment from you talking about your great solution. I am really rooting for you, I want to know what the hell you discovered! hahaha
", "rawMarkdown": "This is the third time I read a comment from you talking about your great solution. I am really rooting for you, I want to know what the hell you discovered! hahaha", "votes": 1 }, { "id": 2917153, "postDate": "2024-07-11T14:07:44.850Z", "content": "Respect! You deserve the championship and the prize.
", "rawMarkdown": "Respect! You deserve the championship and the prize. ", "votes": 1 }, { "id": 2917158, "postDate": "2024-07-11T14:08:21.237Z", "content": "Once upon a time, when \"kernel trick\" still reigned, no one cared about neural networks. Shunning away from NN was so prevalent that if your paper was about NN, your chance of publication would be very low.
\nIn 2012, a dude by the name of Alex Krizhevsky had the balls to use NN and competed, and the rest was history.
\nMoral of the story: Never underestimate someone who has balls.
\nMoral 2 of the story: A small extension to the deadline is a very small price to pay for a chance of a good solution @jerrylin96 intends.
if leaks are allowed and there are no extensions, that would be great.
Are you serious? hahaha. 4 days left.
\nWhy not simply have the person who discovered the leak fill out their home address? We will send the prize and gold medal to you via express delivery.
", "rawMarkdown": "`if leaks are allowed and there are no extensions, that would be great.`\n\nAre you serious? hahaha. 4 days left.\n\nWhy not simply have the person who discovered the leak fill out their home address? We will send the prize and gold medal to you via express delivery. " }, { "id": 2917176, "postDate": "2024-07-11T14:17:03.307Z", "content": "\n\nWhy not simply have the person who discovered the leak fill out their home address? We will send the prize and gold medal to you via express delivery.
\n
Well, since his team already have a prepared leaky solution from previous leak…lol
", "rawMarkdown": ">Why not simply have the person who discovered the leak fill out their home address? We will send the prize and gold medal to you via express delivery.\n\nWell, since his team already have a prepared leaky solution from previous leak...lol", "votes": 2 }, { "id": 2917182, "postDate": "2024-07-11T14:23:00.123Z", "content": "I think so. :) 🥺.
\nNow, I realize that those strange accounts who leak the leakage in the discussion might be friendly to us. They brought leakage into public and start a huge debate. Otherwise, leaky team will go to the top in the last second.
\nThank you. @rkwasny @bethegoose
", "rawMarkdown": "I think so. :) 🥺. \n\nNow, I realize that those strange accounts who leak the leakage in the discussion might be friendly to us. They brought leakage into public and start a huge debate. Otherwise, leaky team will go to the top in the last second.\n\nThank you. @rkwasny @bethegoose ", "votes": 1 }, { "id": 2917191, "postDate": "2024-07-11T14:28:16.363Z", "content": "\n\nNow, I realize that those strange accounts who leak the leakage in the discussion might be friendly to us. They brought leakage into public. Otherwise, leaky team will go to the top in the last second.
\n
I don't know if they had good intentions, but it was definitely good that they revealed the leak. Imagine the horror of reaching the last day and suddenly finding five leaky teams on top. I shudder to think about it.
", "rawMarkdown": "> Now, I realize that those strange accounts who leak the leakage in the discussion might be friendly to us. They brought leakage into public. Otherwise, leaky team will go to the top in the last second.\n\nI don't know if they had good intentions, but it was definitely good that they revealed the leak. Imagine the horror of reaching the last day and suddenly finding five leaky teams on top. I shudder to think about it.", "votes": 3 }, { "id": 2917199, "postDate": "2024-07-11T14:36:58.570Z", "content": "I don't know if they had good intentions
Yeah. I have a hunch that it's a team that already leveraged a leak to score impressively offline, fearing disqualification at the end. Thus, they chose an awkward moment to expose the leak, assuming others wouldn't have time to use the leak to outperform them.
\nAfter the competition closed, they could justify their actions by citing that \"'others' had already published the leak\". And they are very likely to get any penalties.
\nThey just didn't expect such a strong community reaction….
", "rawMarkdown": "`I don't know if they had good intentions`\n\nYeah. I have a hunch that it's a team that already leveraged a leak to score impressively offline, fearing disqualification at the end. Thus, they chose an awkward moment to expose the leak, assuming others wouldn't have time to use the leak to outperform them. \n\nAfter the competition closed, they could justify their actions by citing that \"'others' had already published the leak\". And they are very likely to get any penalties. \n\nThey just didn't expect such a strong community reaction...." }, { "id": 2917201, "postDate": "2024-07-11T14:38:56.767Z", "content": "\n\nImagine the horror of reaching the last day and suddenly finding five leaky teams on top. I shudder to think about it.
\n
This has happened in foursqure competition. Some participants are very good at digging out leaks and hiding it until the last minute.
", "rawMarkdown": ">Imagine the horror of reaching the last day and suddenly finding five leaky teams on top. I shudder to think about it.\n\nThis has happened in foursqure competition. Some participants are very good at digging out leaks and hiding it until the last minute. ", "votes": 2 }, { "id": 2917215, "postDate": "2024-07-11T14:48:19.750Z", "content": "\n\nThis has happened in foursqure competition. Some participants are very good at digging out leaks and hiding it until the last minute.
\n
Yeah…sadly I'm not very good at finding leaks, only at building models :(
", "rawMarkdown": ">This has happened in foursqure competition. Some participants are very good at digging out leaks and hiding it until the last minute.\n\nYeah...sadly I'm not very good at finding leaks, only at building models :(" }, { "id": 2917225, "postDate": "2024-07-11T14:54:00.857Z", "content": "Hahaha, me neither. I just want to finish this competition now and start playing the Elden Ring DLC!
", "rawMarkdown": "Hahaha, me neither. I just want to finish this competition now and start playing the Elden Ring DLC!", "votes": 1 }, { "id": 2917236, "postDate": "2024-07-11T14:56:42.097Z", "content": "\n\nHahaha, me too. I just want to finish this competition now and start playing the Elden Ring DLC!
\n
Forget Elden Ring…you would not find a greater drama than Kaggle drama 🤣
", "rawMarkdown": ">Hahaha, me too. I just want to finish this competition now and start playing the Elden Ring DLC!\n\nForget Elden Ring...you would not find a greater drama than Kaggle drama 🤣" }, { "id": 2917507, "postDate": "2024-07-11T16:51:08.647Z", "content": "extending competition is IMO only a last resort, specifically close to deadlines
\nkaggle requires lots of dedication and people plan their work and vacation around it
I hope the competition can finish on time. Although my ranking is not high, I have made many attempts and I'm a bit tired.
", "rawMarkdown": "I hope the competition can finish on time. Although my ranking is not high, I have made many attempts and I'm a bit tired.", "votes": 2 } ] }, { "id": 2916481, "postDate": "2024-07-11T04:05:08.313Z", "content": "@jerrylin96
\nHere is (relatively) simple, clear and feasible solution for the leak prevention.
\n\nGold medal team must re-submit their solution via time-series api after finalization(no extension).
\n
Agree. But I think there is a possibility to store the data and then process it for inference. And this requires Kaggle to redesign a submission pipeline, which is a bit troublesome for them.
", "rawMarkdown": "> Gold medal team must re-submit their solution via time-series api after finalization(no extension).\n\nAgree. But I think there is a possibility to store the data and then process it for inference. And this requires Kaggle to redesign a submission pipeline, which is a bit troublesome for them. ", "votes": 1 } ] }, { "id": 2919526, "postDate": "2024-07-13T01:43:54.017Z", "content": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it. Maybe i am too romantic or too new into this …
", "rawMarkdown": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it. Maybe i am too romantic or too new into this …", "votes": 6, "replies": [ { "id": 2920030, "postDate": "2024-07-13T11:39:12.023Z", "content": "There is a long tradition of leaks in Kaggle competitions, and in many cases, they have been made public as soon as they were discovered.
\nSome historical examples that come to mind:
\nAs Kagglers, we have repeatedly urged for a solution that includes a (honor) reward for discoverers, which would serve as an incentive to disclose:
\nIn any case, here you can find a video with an interesting perspective by a GrandMaster
\nhttps://www.youtube.com/watch?v=rrWmmqAt6Ts
@jerrylin96
\nI can come up with following choice.
Now I prefer option 1 😭
\nAre there other columns that could cause leaks?
", "rawMarkdown": "Now I prefer option 1 😭\n\nAre there other columns that could cause leaks?" }, { "id": 2916524, "postDate": "2024-07-11T04:38:29.190Z", "content": "For 1 we need to remove also pbuf_SOLIN. And maybe additional columns. And we STILL have the risk of yet another similar leak down the line.
\nFor 3, I hope that it come with an extension so I can utilize leak too…
I preferred 2. Tired……
", "rawMarkdown": "I preferred 2. Tired……", "votes": 2 }, { "id": 2916579, "postDate": "2024-07-11T05:47:03.377Z", "content": "I disagree option 1. This leak is probably intrinsic in the ClimSim dataset. We can't know in advance which feature contains location/timestamp information. I think if host fixed one leak, then someone will find out another leak. It will be endless.
", "rawMarkdown": "I disagree option 1. This leak is probably intrinsic in the ClimSim dataset. We can't know in advance which feature contains location/timestamp information. I think if host fixed one leak, then someone will find out another leak. It will be endless.", "votes": 3 }, { "id": 2916593, "postDate": "2024-07-11T06:00:27.240Z", "content": "Coding check need to cover train part. Only checking inference inputs is not enough.
\nOtherwise, kagglers can use 2d -> 1d models to get oof and then use pseudo labels or distillation to learn this oof.
", "rawMarkdown": "Coding check need to cover train part. Only checking inference inputs is not enough.\n\nOtherwise, kagglers can use 2d -> 1d models to get oof and then use pseudo labels or distillation to learn this oof.", "votes": 4 }, { "id": 2916598, "postDate": "2024-07-11T06:04:34.687Z", "content": "Both of these variables are important for radiative transfer. Ozone has an impact on stratospheric heating rates in particular and insolation is important throughout the column. From a climate modeling perspective you cannot remove them.
", "rawMarkdown": "Both of these variables are important for radiative transfer. Ozone has an impact on stratospheric heating rates in particular and insolation is important throughout the column. From a climate modeling perspective you cannot remove them.", "votes": 5 }, { "id": 2916763, "postDate": "2024-07-11T08:34:28.657Z", "content": "For choice 1, the new dataset should be with irregular timestamp gap(which is constant now), better with a time-series API and another 15 days extension.
\nFor choice 2, it's terrible. It hurts kaggle and host's reputation. It reminds me of bad memories of comps at Chinese Tianchi, I was always thinking some teams cheated but got away with being punished.
\nFor choice 3, another 15 days extension also needed(maybe not enough).
\n\nFor choice 2, it's terrible. It hurts kaggle and host's reputation. It reminds me of some bad memories of comps at Chinese Tianchi, I was always thinking some teams cheated but got away with being punished.
\n
You are right, even though I am the OP who originally requested the host to disqualify. Even if they do it, we will forever suspect that some winners have hidden a cheating method behind their code.
\nSo it will probably be best to extend comp and allow leak.
\nI will be bold and say that even one month. Since with leak it turn to almost a new competition.
\nThe other choice is for the host to produce an extremely irregular and downsampled test, both in time and space (i.e., something like ~50/384 points per timestep). I don't know how much computation it will require. Even then, there is a risk of finding some extraordinary leak.
\nUnfortunately, time series API is useless in this competition since, unlike time series, we want inference to be independent not only of the latter samples but also of the previous ones. For example, with time series API, a user can save the first samples and then slowly build a 'picture' of complete grids and turn it to 2D->1D for the latter samples.
\nSo, all in all, it's probably best to just give us time-space data and turn it into a 2D->1D competition with a massive (as I said, maybe even ~1 month) extension.
\nHonestly, this is what the host should have done from the beginning. As someone said, 'Never underestimate Kagglers.' The more someone tries to bind us to a particular method, the more loopholes we will find. Leading to 'leak competition' instead of 'DL competition' which just ruin the competition for everyone except the few that found leak.
What about timeseries api with the requirement that the inference code be made public/available - much easier to check for irregularities if just dealing with the timeseries api?
", "rawMarkdown": "What about timeseries api with the requirement that the inference code be made public/available - much easier to check for irregularities if just dealing with the timeseries api? \n\n" }, { "id": 2916791, "postDate": "2024-07-11T08:56:57.183Z", "content": "It would be better to extend the deadline for team-up by one week.
\nI don’t want to go solo anymore.
\nI'm really tired. 😂
\n\nI don’t want to go solo anymore.
\n
Nooo I was rooting for you to become a GM :(
\nBut yeah if it turn to 2D->1D comp' with extension I don't know if I will have the energy to remain solo lol
\n\nWhat about timeseries api with the requirement that the inference code be made public/available - much easier to check for irregularities if just dealing with the timeseries api?
\n
'Never underestimate Kagglers'- SOMEONE will probably find a loophole or succeed to hide his cheating 🤣
", "rawMarkdown": ">What about timeseries api with the requirement that the inference code be made public/available - much easier to check for irregularities if just dealing with the timeseries api?\n\n'Never underestimate Kagglers'- SOMEONE will probably find a loophole or succeed to hide his cheating 🤣", "votes": 2 }, { "id": 2916808, "postDate": "2024-07-11T09:06:41.637Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2916809, "postDate": "2024-07-11T09:06:58.063Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2916810, "postDate": "2024-07-11T09:09:47.823Z", "content": "This goes along with the advice I gave here earlier: 😀
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519249#2916364
\nBut I think coding check should need to cover train part. as @hookman said.
All top teams now have good reputation, only few teams would take that risk.
\nalso concat with irregular historical data can not boost much score and is time-consuming both in training and inference phase.
\nalso inference code is much easier to check for us than rerun all their training code.
\nso i think this choice is fine!
\nhidden test and time series API are needed.
\n\nAll top teams now have good reputation, only few teams would take that risk.
\n
'All top teams now have good reputation,'
\nFor now. But with leak you cannot know how many with less reputation will jump to top 10…
\nBut maybe the situation is still salvageable indeed with irregular test+time-series API+public inference code. You are making strong points.
I think 2) is not possible as many will use the leak implicitly. It has happened many times before that eg larger and more complex models were beneficial for leaky data as they can capture the leak better.
", "rawMarkdown": "I think 2) is not possible as many will use the leak implicitly. It has happened many times before that eg larger and more complex models were beneficial for leaky data as they can capture the leak better.", "votes": 4 } ] }, { "id": 2915373, "postDate": "2024-07-10T13:21:09.903Z", "content": "@shlomoron Well, this discussion's opinion seems contradict to your previous comment. Have you change your mind?
\n\n\nKaggle is very simple, really. EVERYTHING is allowed…except what is written in the Rules tab.
\n
\n Does anything in said tab say, 'multi-row-to-single-target approach is forbidden'?
\n No. Hence, it is allowed.
https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/514626#2888661
", "rawMarkdown": "@shlomoron Well, this discussion's opinion seems contradict to your previous comment. Have you change your mind?\n\n> Kaggle is very simple, really. EVERYTHING is allowed…except what is written in the Rules tab.\n> Does anything in said tab say, 'multi-row-to-single-target approach is forbidden'?\n> No. Hence, it is allowed.\n\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/514626#2888661", "votes": 6, "replies": [ { "id": 2915376, "postDate": "2024-07-10T13:24:40.863Z", "content": "If this leak was not revealed before competition end, I would not be able to say anything. Since it is allowed by the rules.
\nSince it was found before the end of the competition, I can ask for the host to 'soildify' their 'spirit of the competition' in the rules, as it became relevant again.
\nNo contradictions here imho.
yes but in the meantime the hosts made clear that multi-row-to-single-target is not allowed.
", "rawMarkdown": "yes but in the meantime the hosts made clear that multi-row-to-single-target is not allowed.", "votes": 1 }, { "id": 2915479, "postDate": "2024-07-10T14:29:59.760Z", "content": "This discussion thread of @shlomoron (known as greySnow) does not contradict his/her earlier comment when greySnow said \"EVERYTHING is allowed…except what is written in the Rules tab.\"
\nIt is one thing how one interprets the exact definition of Rules with respect to what're technically allowed. It is entirely another thing whether one decides to do it or not (based possibly on moral or other grounds). greySnow and I may had some contention about the former, but certainly not the latter.
", "rawMarkdown": "This discussion thread of @shlomoron (known as greySnow) does not contradict his/her earlier comment when greySnow said \"EVERYTHING is allowed…except what is written in the Rules tab.\"\n\nIt is one thing how one interprets the exact definition of Rules with respect to what're *technically* allowed. It is entirely another thing whether one decides to do it or not (based possibly on moral or other grounds). greySnow and I may had some contention about the former, but certainly not the latter.", "replies": [ { "id": 2915503, "postDate": "2024-07-10T14:42:38.703Z", "content": "\n\ngreySnow and I may had some contention about the former, but certainly not the latter.
\n
Thank you for your support, but we actually are also contested on the latter 😅
\nI think it is entirely moral to utilize leak…
\nBut since the leak was revealed, I can ask for better rules since it's in my best interest. (Well, and in the best interest of most participants, though most certainly not ALL.)
\nThis way or another, it's up to the host. Nothing moral or immoral either way.
\nAnd my techniques will lend me gold, if not in this competition, then in the next. So it's not the end of the world even if leak is used…
I knew this was coming. That's why I added \"moral or other grounds\" 😅
\nAnd that \"other grounds\" could certainly be one's best interest. Not a problem at all.
\nSo, I'm certainly correct in saying that we're not contested in the latter.
\nAnd this post of yours certainly has zero contradiction to your earlier comment.
", "rawMarkdown": "I knew this was coming. That's why I added \"moral or **other grounds**\" 😅\n\nAnd that \"other grounds\" could certainly be one's best interest. Not a problem at all.\n\nSo, I'm certainly correct in saying that we're not contested in the latter.\n\nAnd this post of yours certainly has zero contradiction to your earlier comment.", "votes": 1 }, { "id": 2915544, "postDate": "2024-07-10T14:55:43.020Z", "content": "Hahaha you got me there
", "rawMarkdown": "Hahaha you got me there", "votes": 1 } ] } ] }, { "id": 2915991, "postDate": "2024-07-10T18:03:42.240Z", "content": "Just two cents from a Kaggle newbie who has little skin in the game time-investment-wise: it seems to me (but feel free to argue!) that solutions utilizing this leak will be useless to the host/climate community. As far as I can see, the reason that pbuf_ozone gives the location/time-stamp is just because these values have been fixed to constants in the individual E3SM-MMF simulations. It's not because ozone is truly a predictor of location/time in the real world. The ClimSim paper explicitly states that this \"limitation\" will be improved in subsequent models. Once it has been fixed, any model that assumes all data-points with the same pbuf_ozone are related, and uses many to predict one, may not just lose their advantage, they may perform significantly worse than the one-to-one solutions.
\nTherefore, if the prizes are supposed to go to those who produce something useful for the host (which seems reasonable to me), it would not be unfair to disqualify the leak-based solutions.
", "rawMarkdown": "Just two cents from a Kaggle newbie who has little skin in the game time-investment-wise: it seems to me (but feel free to argue!) that solutions utilizing this leak will be useless to the host/climate community. As far as I can see, the reason that pbuf_ozone gives the location/time-stamp is just because these values have been fixed to constants in the individual E3SM-MMF simulations. It's not because ozone is truly a predictor of location/time in the real world. The ClimSim paper explicitly states that this \"limitation\" will be improved in subsequent models. Once it has been fixed, any model that assumes all data-points with the same pbuf_ozone are related, and uses many to predict one, may not just lose their advantage, they may perform significantly worse than the one-to-one solutions.\n\nTherefore, if the prizes are supposed to go to those who produce something useful for the host (which seems reasonable to me), it would not be unfair to disqualify the leak-based solutions.", "votes": 3 }, { "id": 2915682, "postDate": "2024-07-10T15:54:17.820Z", "content": "Not sure a disqualification is a fair response given it's so close to the end of the comp - seems unfair for those who found it independently and managed to make use of it…
\nFWIW - we aren't using it.
", "rawMarkdown": "Not sure a disqualification is a fair response given it's so close to the end of the comp - seems unfair for those who found it independently and managed to make use of it...\n\nFWIW - we aren't using it.", "votes": 1, "replies": [ { "id": 2915693, "postDate": "2024-07-10T15:58:56.050Z", "content": "It's unfair either way. If host does not disqualifiy, it's unfair for all teams the invested a lot of effort in good faith after host stressed that this would be 1D->1D and even changed test set and extended the competition for this…but fairness is just a word. The world is unfair and it's ok lol.
", "rawMarkdown": "It's unfair either way. If host does not disqualifiy, it's unfair for all teams the invested a lot of effort in good faith after host stressed that this would be 1D->1D and even changed test set and extended the competition for this...but fairness is just a word. The world is unfair and it's ok lol.", "votes": 4 } ] }, { "id": 2915093, "postDate": "2024-07-10T10:27:21.647Z", "content": "Completely agree.
\nBut this means that organizers need additional energy and technology to detect these leak-based solutions, and there may be some omissions.
", "rawMarkdown": "Completely agree. \n\nBut this means that organizers need additional energy and technology to detect these leak-based solutions, and there may be some omissions.", "votes": 1, "replies": [ { "id": 2915111, "postDate": "2024-07-10T10:38:56.493Z", "content": "True, it would require energy and effort. I can only request.
", "rawMarkdown": "True, it would require energy and effort. I can only request.", "votes": 1 } ] }, { "id": 2915679, "postDate": "2024-07-10T15:53:00.243Z", "content": "I understand that being #1 on public LB, this isn't good news, but that is the nature of Kaggle competitions. It is the Optiver competition from 2021 all over again:
\nhttps://www.kaggle.com/competitions/optiver-realized-volatility-prediction/discussion/284121
It was a very similar situation, except that only 10-20 teams used the leak and it wasn't made public. Admins made changes to the data to effectively disqualify these teams. However, this created a lot of backlash in the community and they ultimately decided to revert the change, allowing the leak (the leak was much, much worse/ gave a massive boost). Now there is a small difference in that the leak was made public a few days before the end of the competition, but disqualifying leak solutions would also make it unfair for teams who assumed everything is fair game and have used the leak all the way. Either way, I think it isn't great and an extension of the timeline would be the least bad thing the hosts could do in my opinion (if they can afford that).
", "rawMarkdown": "I understand that being #1 on public LB, this isn't good news, but that is the nature of Kaggle competitions. It is the Optiver competition from 2021 all over again:\nhttps://www.kaggle.com/competitions/optiver-realized-volatility-prediction/discussion/284121\n\nIt was a very similar situation, except that only 10-20 teams used the leak and it wasn't made public. Admins made changes to the data to effectively disqualify these teams. However, this created a lot of backlash in the community and they ultimately decided to revert the change, allowing the leak (the leak was much, much worse/ gave a massive boost). Now there is a small difference in that the leak was made public a few days before the end of the competition, but disqualifying leak solutions would also make it unfair for teams who assumed everything is fair game and have used the leak all the way. Either way, I think it isn't great and an extension of the timeline would be the least bad thing the hosts could do in my opinion (if they can afford that).", "votes": 2, "replies": [ { "id": 2915741, "postDate": "2024-07-10T16:14:28.603Z", "content": "\n\nunfair for teams who assumed everything is fair game and have used the leak all the way
\n
After the hosts made his point time and again about the 'spirit of the competition', including changing the test set once and extending the competition (without any backlash, mind you), I would say that it's hardly unfair. More like 'professional risk' that leak utilizers took knowingly…but it's up to the host. And yes, if he does not disqualify, I at least hope for an extension to be able to utilize the leak properly, too. Even tho I'm tired, lol
", "rawMarkdown": ">unfair for teams who assumed everything is fair game and have used the leak all the way\n\nAfter the hosts made his point time and again about the 'spirit of the competition', including changing the test set once and extending the competition (without any backlash, mind you), I would say that it's hardly unfair. More like 'professional risk' that leak utilizers took knowingly...but it's up to the host. And yes, if he does not disqualify, I at least hope for an extension to be able to utilize the leak properly, too. Even tho I'm tired, lol", "votes": 2, "replies": [ { "id": 2915789, "postDate": "2024-07-10T16:38:33.017Z", "content": "As Jerry Lin said:
\n\n\nKnowing this, it is fairly obvious that the intention of the competition designers was to not grant you access to this information. As Kaggle user @ryches puts it, using this information indeed goes against the spirit of the competition.
\n
This doesn't sound like a rule which is enforced upon the participants to me. Similar things were said in the Optiver competition when there was suspected there might be a leak.
\nBut in the end, the Kaggler is in a tough spot; utilize the leak if they found it (since why not, every other winning team uses some kind of leak in their solution, and how else to be competitive if other top teams use it), or not use it, possibly ending in a bad place on the LB even though the solution was great. Or be so courageous to make the leak public (and I don't mean a few days before end). Anyway, given how widespread and frequent the utilization of leaks is in kaggle competitions (and I think it is clear that this is almost always against the hosts intention and not in the spirit of the competition), it would be a strange exception to make.
\nI think it would be fine if Kaggle would clearly state in the beginning of a competition, that such solutions will be disqualified, but AFAIK this wasn't the case here. Also it is hardly detectable for non-prize winning solutions.
\nI also only heard about this leak a few hours ago. But of course it makes you think, if top teams are now required to get a solution out there which makes use of the leak in a final sprint ?
I agree that leak utilization is OK. Ultimately, it's not a question of fairness but what is best for the host. I can only make my request in my (obviously) best interest. Since rule change by last 5 days is withing the rules, and align with host's intention, I don't think there is a problem with that, even though it's 'unfair for those who utilized leak' (knowing that there is a chance that leak is leaked before comp' end leading to their efforts wasted).
", "rawMarkdown": "I agree that leak utilization is OK. Ultimately, it's not a question of fairness but what is best for the host. I can only make my request in my (obviously) best interest. Since rule change by last 5 days is withing the rules, and align with host's intention, I don't think there is a problem with that, even though it's 'unfair for those who utilized leak' (knowing that there is a chance that leak is leaked before comp' end leading to their efforts wasted)." }, { "id": 2915873, "postDate": "2024-07-10T17:20:42.370Z", "content": "For the record, let me say this: I will stick with not using any leak and genuinely treat this as an i.i.d. regression problem. All I ask of @jerrylin96 is one more week (for a deadline extension to Jul 22) so I can use every last drop of my brain juice to try to remain competitive even against approaches using the leak.
\nConsider this an official contract signed with my soul.
", "rawMarkdown": "For the record, let me say this: I will stick with *not* using any leak and genuinely treat this as an i.i.d. regression problem. All I ask of @jerrylin96 is one more week (for a deadline extension to Jul 22) so I can use every last drop of my brain juice to try to remain competitive even against approaches using the leak.\n\nConsider this an official contract signed with my soul." } ] } ] }, { "id": 2915077, "postDate": "2024-07-10T10:17:37.687Z", "content": "Totally agree with you and can't wait for reading your solution
", "rawMarkdown": "Totally agree with you and can't wait for reading your solution", "votes": 2 }, { "id": 2915110, "postDate": "2024-07-10T10:38:55.167Z", "content": "Totally agree!
\nI suggest top 50 teams to make a promise here that your team will not use this trick.😀
Hahaha for silver range there is probably not much to do. It's too many solutions to keep track on.
\nBut for gold range (and surly for price) it's possible.
nice tutorial
", "rawMarkdown": "nice tutorial", "votes": -2 }, { "id": 2920981, "postDate": "2024-07-14T01:17:19.867Z", "content": "What do you mean by 2D > 1D? What does 2D mean?
", "rawMarkdown": "What do you mean by 2D > 1D? What does 2D mean?" }, { "id": 2920108, "postDate": "2024-07-13T12:37:27.537Z", "content": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it
", "rawMarkdown": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it" }, { "id": 2919221, "postDate": "2024-07-12T18:37:06.683Z", "content": "Is the timestamp/district for new test set perfectly recovered?
", "rawMarkdown": "Is the timestamp/district for new test set perfectly recovered? \n\n" }, { "id": 2918240, "postDate": "2024-07-12T05:31:49.660Z", "content": "hello good outcome
", "rawMarkdown": "hello good outcome" }, { "id": 2916434, "postDate": "2024-07-11T03:00:09.920Z", "content": "Is the timestamp/district for new test set perfectly recovered? 🤔
", "rawMarkdown": "Is the timestamp/district for new test set perfectly recovered? 🤔\n" }, { "id": 2916250, "postDate": "2024-07-10T20:47:24.640Z", "content": "Thanks for the suggestion! We are looking into this.
", "rawMarkdown": "Thanks for the suggestion! We are looking into this." }, { "id": 2916436, "postDate": "2024-07-11T03:02:38.087Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 2921801, "postDate": "2024-07-14T16:56:15.587Z", "content": "Thanks for the nice information
", "rawMarkdown": "Thanks for the nice information" } ], "comments": [ { "id": 2917435, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-11T16:24:06.587000", "content": "From Section B.10 :
\n\"Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.
\nA disqualified participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.\"
\nWe will disqualify leak-based solutions for cash prizes. To be eligible for a cash prize, Kaggle teams will have to verify their training and inference code with the competition hosts.
", "votes": 28, "replies": [ { "id": 2917481, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T16:42:11.597000", "content": "Not to undermine this decision, and honestly it is the best possible outcome for me.
\nI just want to ask, are you sure this task is feasible given the complexities involved?
\nSome teams might use ensembles of tens of models, each requiring heavy computing and long training time, using complex pipelines involving pseudo-labeling, distillation, etc. Easy to hide something behind a lot of compute and complex code. I don't want to lose to some good-at-hiding team; in this case, I would prefer to have an extension and leak allowed and train leaky solution (from scratch, in my case).
\np.s. Surprisingly (or not?) I believe my own solution would be easy to verify, lol.
You could technically build a pipeline to get leak based predictions for the test-set, and only use these predictions to validate models and new ideas during training. I can imagine this could give a small boost but would not be detectable. I think there are still many ways you can gain an unfair advantage which are hard to catch
", "votes": 1, "replies": [] }, { "id": 2917503, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T16:50:12.493000", "content": "There are many Masters and Grandmasters in top teams. I think they won't give a try to use leak at risk of totally losing their reputation in kaggle.
", "votes": 1, "replies": [] }, { "id": 2917525, "author_name": "Max2020", "author_url": "", "post_date": "2024-07-11T16:55:46.543000", "content": "To alleviate the pressure on the host to conduct checks independently, I propose that all award candidates be required to open-source all their training codes, models, and inference scripts. This way, everyone can inspect each other's work and even conduct random checks.
", "votes": 0, "replies": [] }, { "id": 2917528, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-11T16:57:29.860000", "content": "Hi @shlomoron , considering that most of the top-ranked teams are Grandmasters or Masters on the verge of becoming Grandmasters, I believe the risk of them hiding leaks is minimal. I trust that the vast majority of Grandmasters, along with the masters hoping to reach gm, care deeply about their reputation within the community. It wouldn't be worth gambling for this single competition with their GM's reputation.
", "votes": -1, "replies": [] }, { "id": 2917540, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T16:59:55.937000", "content": "\n\nThere are many Masters and Grandmasters in top teams. I think they won't give a try to use leak at risk of totally losing their reputation in kaggle.
\n
Well, there were cases of kaggle GMs cheating so everything is possible…
", "votes": 1, "replies": [] }, { "id": 2917542, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T17:01:53.677000", "content": "That's exactly where risk comes from. They might quit kaggle, even lose jobs.
", "votes": 1, "replies": [] }, { "id": 2917544, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T17:03:45.093000", "content": "\n\nTo alleviate the pressure on the host to conduct checks independently, I propose that all award candidates be required to open-source all their training codes, models, and inference scripts. This way, everyone can inspect each other's work and even conduct random checks.
\n
On the contrary, the top 10 mustn't publish their solutions in the first days.
\nImagine this scenario: Team X uses cheating, reads Team Y's legit solution on the forum, and then incorporates Team Y's ideas into his 'solution' that he sends to the host to verify. With new ideas, his score becomes legit.
\nOf course, after host verified it would be good to have a second verification on the public forum.
Your consideration is very thorough. However, we are essentially dealing with a game theory scenario. If the host decides to conduct a thorough review, the likelihood of individuals taking risks will significantly decrease. Considering this, I believe that it would be more fair and economically approach?
", "votes": 2, "replies": [] } ] }, { "id": 2917504, "author_name": "bethegoose", "author_url": "", "post_date": "2024-07-11T16:50:13.643000", "content": "Under which of those criterion are competitors going to be disqualified? Using the data at hand is not cheating.
\nA competition is about taking a pile of data, finding some signal, and generating value. The rules were such that the provided dataset (and more!) were encouraged to be used to make the best possible predictions. While there was indication that the organizers did not want certain types of solutions to be used, those were never formally specified. Multiple questions for clarifications in the forums went ignored. When the dataset was updated, that was the time to make a formal rule change, if required.
\nKaggle: use data, find signal, but thread the needle on which signals are acceptable to use without official codified guidance.
", "votes": -11, "replies": [ { "id": 2917519, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T16:53:36.700000", "content": "Hey, Please use your main account to talk, and do not use a secondary account to ask questions.
", "votes": 15, "replies": [] }, { "id": 2917631, "author_name": "bethegoose", "author_url": "", "post_date": "2024-07-11T17:45:55.870000", "content": "This is my only account.
", "votes": -10, "replies": [] }, { "id": 2918040, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-12T01:27:42.740000", "content": "Ok, I notice you have really deep undetstanding about this dataset, so why not give some submits and hava fun?
\nAlso, could you please explain why you leaked leakage in last 5 days and what is your intention?
\nDude, your account behaviou is really strange and I can't help but suspect your are Oliver Queen,the Green Arrow, possessing two faces.
\nimposter.png
\n
Looks like everyone is running out of patience.
\nI think there will be at least several months to verify top teams' work one by one.
\nif this's final decision, i suggest top 5 teams to public their training and inference code, otherwise it's really terrible.
Copying from my other comment: ' top 10 mustn't publish their solutions in the first days.
\nImagine this scenario: Team X uses cheating, reads Team Y's legit solution on the forum, and then incorporates Team Y's ideas into his 'solution' that he sends to the host to verify. With new ideas, his score becomes legit.
\nOf course, after host verified it would be good to have a second verification on the public forum.'
It makes sense!
", "votes": 3, "replies": [] }, { "id": 2918165, "author_name": "NoviceLearner", "author_url": "", "post_date": "2024-07-12T04:03:35.317000", "content": "Smart choice is to avoid publishing solution until the end of code verification.
", "votes": 0, "replies": [] } ] }, { "id": 2917692, "author_name": "Ryota", "author_url": "", "post_date": "2024-07-11T18:19:24.557000", "content": "\nThank you for clarifying the measures to be taken when leaks are used. However, the criteria for what falls under a leak are still unclear, and I would like this to be clarified.
\nFrom your intentions, we understand that using explicit leaks present in the test data (such as pbuf_ozone_2) to predict a single row from multiple rows is prohibited.
\nHowever, for example:
cam_in_LANDFRAC alone is sufficient to reveal the identity of 163 locations. Is anything which gives spatial temporal information allowed?
I understand that the provided features, such as cam_in_LANDFRAC, are sufficient to give the model information about the location. Therefore, I will not include such features in future training. However, what if such features were included in early experiments and left as they were? In this case, there is not enough time to retrain the model according to the rules.
\nAs a solution, those with and without these features are essentially the same, aligning with the host's intentions. However, if this is considered a violation under the newly established rules, will those teams be banned? What I want to clarify is the rules.
On the one hand, this has opened up a new way of dealing with leaks; in the future, competitions with leaks may follow this example, which is the most worrying aspect. On the other hand, the time spent cooperating with the organizers to fully restore the code may not be less than the time we spend re-running the model. Sometimes, due to different environments and configurations, it might not run at all.
", "votes": 3, "replies": [] } ] }, { "id": 2916872, "author_name": "Vasilis", "author_url": "", "post_date": "2024-07-11T10:19:50.783000", "content": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it. Maybe i am too romantic or too new into this …
", "votes": 9, "replies": [] }, { "id": 2916314, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-10T22:31:07.513000", "content": "Hello all. I am waiting to confirm this with the Kaggle staff, but it may be safe to assume for now that we will disqualify any submissions that take advantage of any leak or use a multicolumn approach for cash prizes and gold medals.
", "votes": 12, "replies": [ { "id": 2916321, "author_name": "Camaro", "author_url": "", "post_date": "2024-07-10T22:38:01.803000", "content": "Thank you for the answer! Let me clarify. Will it be applied only for the test time? As discussed in other threads, some training techniques unintentionally use other sample information. I believe disqualifying such a solution is not what you intend, but clarification would be better.
", "votes": 2, "replies": [ { "id": 2916323, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-10T22:45:49.803000", "content": "Can you explain what you mean by unintentionally use other sample information via an explicit example?
", "votes": 1, "replies": [] }, { "id": 2916328, "author_name": "Camaro", "author_url": "", "post_date": "2024-07-10T23:01:01.620000", "content": "Typical one is batch normalization. Other example is pseudo labeling.
\nDisucussed in the thread.
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519233
And how about explicit use of multi columns information at training time? Like pretraining multi columns model, then fine tuning for single columns setting. Or using nearest columns target as auxiliary loss and so on… (I don't use these though.)
", "votes": 1, "replies": [] }, { "id": 2916334, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-10T23:16:36.113000", "content": "Typically by using other sample information, I am referring to using multiple samples corresponding to the same timestep. It would be difficult to do this unintentionally.
\nWe will likely seek to disqualify submissions that reverse-engineer location / timestep information at any point.
", "votes": 3, "replies": [] }, { "id": 2916338, "author_name": "Bilzard", "author_url": "", "post_date": "2024-07-10T23:30:12.687000", "content": "\n\nTypically by using other sample information, I am referring to using multiple samples corresponding to the same timestep. It would be difficult to do this unintentionally.
\n
@jerrylin96
\nTypical machine learning model easy to learn location and timestamp information from pbuf_ozon_2 or other targets it may encode them. That means all participants model more or less utilizes this “leak” if participants know or not. It’s no exception like the baseline model the host prepared. How do you think of that fact?
\n\nTypical machine learning model easy to learn location and timestamp information from pbuf_ozon_2 or other targets it may encode them.
\n
I totally agree.
\nEven if utilization of the specific information like pbuf_ozon_2 will be explicitly prohibited, ML models can learn the timestamp / location info somewhat, because we know how to create training dataset.
I suggest a criteria like below,
\nThis competition is a CSV submission competition, it is quite challenging to investigate whether there has been any leaking solution. Additionally, since the resource consumption of this competition lies in training rather than inference, I suggest the following:
\nThe suggestion proposed by @hengzheng is reasonable. @jerrylin96
", "votes": 7, "replies": [] }, { "id": 2916596, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T06:01:28.023000", "content": "IMHO, Coding check need to cover train part. Only checking inference inputs is not enough.
\nOtherwise, kagglers can use 2d -> 1d models to get oof and then use pseudo labels or distillation to learn this oof.
", "votes": 1, "replies": [] } ] }, { "id": 2916394, "author_name": "charmq", "author_url": "", "post_date": "2024-07-11T01:24:52.700000", "content": "I declare that I will use the leak unless the competition host makes the rules clear to prohibits the leak. (how about pseudo-labeling? stacking? distilation? batch normalization? any other trick? etc). In a competition, there is no choice but to not spend the time to do EVERYTHING that are necessary to win. If the competition host changes the rules, I will follow the rules, but the new rules should be clear and feasible (to hold fairness.)
\nThe recently concluded Image Matching Challenge 2024 was also won by solutions that were not intended by the host. In some test cases, the problem was solved by estimating the camera order without directly solving the camera pose estimation that should have been solved. There were probably many other competitions where solutions were used that the host did not intend or want.
\nAnyway, I am tired. I want to be free from this competition as soon as possible.
\nActually, we spent not a short period of time (we are concentrating on one or two weeks. we have already found leak independently of yesterday's leak revelations from someone.) building the model utilizing timestamp information because it is not prohibited by rules and any rule change that prohibits certain methods is unfair, at least to us.
", "votes": 3, "replies": [ { "id": 2916399, "author_name": "heng", "author_url": "", "post_date": "2024-07-11T01:45:27.337000", "content": "Foursquare again?
", "votes": 4, "replies": [] }, { "id": 2916401, "author_name": "charmq", "author_url": "", "post_date": "2024-07-11T01:51:28.027000", "content": "The situations are similar but different. My stomach pain is the same.
", "votes": -1, "replies": [] }, { "id": 2916412, "author_name": "kami", "author_url": "", "post_date": "2024-07-11T02:24:20.863000", "content": "I completely agree with @charmq 's opinion.
\nIf the rules are clarified, I will follow them as well. However, there are no rules prohibiting the use of location or time, including the multi-column to single-column approach. Given that it is difficult to clearly distinguish between prohibited and non-prohibited methods, adding rules that everyone can understand is unrealistic. If these methods are to be prohibited, the competition should be designed from the outset to make their use impossible, such as by using time series API.
Furthermore, if prohibitions are introduced, the ambiguity of the rules could lead to different interpretations, potentially undermining fairness among competitors. We are simply doing our best within the bounds of the current rules, and it is unfair to deny us for that.
", "votes": -2, "replies": [] }, { "id": 2916414, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T02:27:33.223000", "content": "If a leak is discovered for the first time, I can understand. But it is the second time, so I cannot understand at all.
\nThe host has emphasized many times, not to use 2d -> 1d solution and the host even changed dataset for this reason. I have no idea why you guys still keep digging this, as you said \"not a short period of time\" and \"we have already found leak independently of yesterday's leak revelations from someone.\".
\nIf you said you used leak because of yesterday's leak revelations from someone, I totally understand
\nWhat confused me is that before yesterday's leak revelations from someone and after the host changed dataset, you still spend 2 weeks to dig leak
\nI don't think you are tired. Instead, I think you are full of energy, you have one or two weeks to dig something forbidden.
\nREALLY confused. :(
", "votes": 22, "replies": [] }, { "id": 2916417, "author_name": "kami", "author_url": "", "post_date": "2024-07-11T02:34:33.443000", "content": "It's very unfortunate and regrettable, but the host's intention and the competition rules are different things.
\nIf someone does their best within the rules, we also need to do our best to compete.
I think same thing as @kami634 's post. (https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519249#2916412)
\n\n\nGiven that it is difficult to clearly distinguish between prohibited and non-prohibited methods, adding rules that everyone can understand is unrealistic.
\n
It's true that making new rules itself is too difficult and unrealistic.
\n\n\nWe are simply doing our best within the bounds of the current rules, and it is unfair to deny us for that.
\n
If the rules are changed, we are forced to spend time to modify the model to comply with the changes. And the host's acceptance of the model itself is not known until the end (because the criteria are vague).
", "votes": 0, "replies": [] }, { "id": 2916426, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-11T02:44:10.560000", "content": "A project will last 3 months
\n------------------- The early stage of the project ---------------
\nSomeone1: \"Can we do that? \"
\nBoss: \"No, we don't want that. \"
------------------- The middle stage of the project ---------------
\nSomeone2: \"We have successfully do that.\"
\nBoss: \"No, we don't want that. Since there is no much time left, let's postpone this project for 2 weeks\"
------------------- The last 5 days of the project ---------------
\nSomeone3: \"The rule of this project doesn't forbid this. So we still spend much time on it. Although you don't want that, you have to give us high kpi(high rank) and pay us money. \"
\nBoss:….
Hosts pay money for a solution useful for them. In this case, they pay money for human progress. Since they don't want the solution (And they did repeatedly stress it, again and again and again), I think they at least have the right to refuse to pay the money.
", "votes": 10, "replies": [] }, { "id": 2916427, "author_name": "charmq", "author_url": "", "post_date": "2024-07-11T02:45:55.183000", "content": "\n\nhost's intention and the competition rules are different things
\n
I thought this is true. If the host explicitly forbidden it earlier in the rules, we did work on another approach.
", "votes": -2, "replies": [] }, { "id": 2916437, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-11T03:04:38.420000", "content": "But they did repeatedly say that they don't want that. And they had postponed this competition for this issues 2 weeks. Still, you think it should be allowed.
", "votes": 5, "replies": [] }, { "id": 2916440, "author_name": "kami", "author_url": "", "post_date": "2024-07-11T03:08:48.717000", "content": "\n\nthey did repeatedly say that they don't want that
\n
However, the host did not answer anything about the clear prohibition in the rules.
\nTherefore, the interpretation is divided. If we could have clearly heard that it was prohibited, we would have explored a different approach.
\nIf it were to be prohibited, it would not have been necessary to replace the test set…
They can't check all of the messages on the forum. But they have replaced the test set and postpone it for 2 weeks, isn't it clear enough?
", "votes": 2, "replies": [] }, { "id": 2916443, "author_name": "heng", "author_url": "", "post_date": "2024-07-11T03:14:36.647000", "content": "So if there hadn't been yesterday's reveal post, were you guys planning to hide until the last day or even the last minute, like in the Foursquare?
", "votes": 8, "replies": [] }, { "id": 2916444, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-11T03:20:30.870000", "content": "The original top two teams in Deepfake Challenge didn't clearly violate the rules, but they were nonetheless demoted or disqualified in the end. In fact, I think what happened here is far worse than anything in the Deepfakes. So I won't be surprise if the host decide to disqualified the prize and even rank.
", "votes": 0, "replies": [] }, { "id": 2916445, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T03:20:31.887000", "content": "I totally agree with host's intention and the competition rules are different things.
If you said you used leak because of yesterday's leak revelations from someone, I totally understand
\nWhat confused me is that before yesterday's leak revelations from someone and after the host changed dataset, you still spend 2 weeks to dig leak.
\nActually, I have explored the leak before dataset change. My method is similar but not exactly same with leak revelations from someone. I use t-sne to plot the 60 pbuf_ozone features group. A year data make a circle. Point order is exactly time order. But since host said not to use time order, I didn't spend time digging this after dataset change. It's meaningless.
@hookman
\nYou found the potential of the leak but did not use it. I respect that attitude of yours. (I also learned a lot from your use of t-SNE)
\nI understand host's intenttion. On the other hand, kaggle is a competition. There were several questions in discussion, and I also asked questions directly via email, but they were ignored.
\nAs long as the host did not explicitly state that it is forbidden, we have to consider hacking the leak in order to win the competition.
you were proud of using leak and you just think you fellow rules。Have you no shame?
", "votes": -8, "replies": [] }, { "id": 2916497, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T04:19:07.820000", "content": "\nTo some extent, I agree with you. Although I doubt intention of digging leak, everyone has different thoughts and it's exactly grey area. I can not judge too much.
\nIt's true that the host did not explicitly state that it is forbidden. But I think even if the host said it's forbidden, it's though very hard to check this. The training is quiet time-consuming and host obviously cannot check everyone's code.
\nLet's just wait and see how host deal with this.
", "votes": 5, "replies": [] }, { "id": 2916695, "author_name": "Yusef A.", "author_url": "", "post_date": "2024-07-11T07:25:52.230000", "content": "I think I would have agreed with this take - but the host just said in the post that you replied to that it wouldn't be permissible..
", "votes": 0, "replies": [] }, { "id": 2916862, "author_name": "Jacoby Jaeger", "author_url": "", "post_date": "2024-07-11T10:14:45.607000", "content": "\n\nIf a leak is discovered for the first time, I can understand. But it is the second time, so I cannot understand at all.
\n
This being the second leak only proves the host's negligence. These leaks should be avoided by the competition design and not by additional rules that can only be enforced arbitrarily. Careful design could have prevented this from the beginning and since they weren't careful enough the first time, the first leak should have been a wakeup call. It's not unreasonable to expect that hosts check that no features map to any parameters they are trying to keep hidden; it should be a mandatory step in setting up any competition really. We've had a string of competitions recently sabotaged by kaggle and the hosts not making the simplest tests and leaving competitions vulnerable.
", "votes": 0, "replies": [] } ] }, { "id": 2916501, "author_name": "Sijun Xu", "author_url": "", "post_date": "2024-07-11T04:21:42.060000", "content": "Will the competition be ended soon, or will keep the ddl unchanged?
", "votes": 0, "replies": [ { "id": 2916540, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-11T04:50:09.317000", "content": "It's impossible to end immediately. But disqualified the rank and money is possible.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2915289, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2024-07-10T12:32:56.197000", "content": "New Competition Idea: Predict the next deadline extension
", "votes": 8, "replies": [ { "id": 2915323, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-10T12:50:52.013000", "content": "We are sooo tired tho 😭
", "votes": 4, "replies": [] } ] }, { "id": 2916993, "author_name": "kurupical", "author_url": "", "post_date": "2024-07-11T12:09:23.783000", "content": "@jerrylin96
\nWhatever you decide, I hope you will not extend the competition. You previously referred to the 2 week extension as \"good news\", but it is clearly \"bad news\" as evidenced by participents reactions.
\nWe kagglers are more committed to kaggle than you might think. We are working as hard as we can towards the end of the competition period. We have a schedule that is tailored to the competition.
\nIn the past, when the competition was extended, some people said \"I was planning a trip right after the competition was over\"(*1)
Extending this further would be like running a full marathon, reaching the 40 km mark, and starting the final sprint, only to announce that the finish line has been changed to 50 km.
\n(*1) https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/110777#637593
", "votes": 6, "replies": [ { "id": 2917001, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T12:19:05.393000", "content": "I would have to disagree here. It's better to have an extended good competition than a finished-in-time bad one. We will just need to stay strong and persevere.
", "votes": 8, "replies": [ { "id": 2917034, "author_name": "kurupical", "author_url": "", "post_date": "2024-07-11T12:38:10.987000", "content": "\n\nelse: nothing has changed in the situation, so no extension is needed regardless of whether the host allows the use of leaks or not
\n
A LOT has changed. If leak is allowed, it's in the best interest of both most competitors (who only now know of leak) and the host (who want to get the best possible solutions, whether based on leak or not) to extend. The ONLY people with whom an extension would not be in their interest is the tiny minority (your team included, mind you) that already have a leak-based solution. It's a VERY different situation from what you referred to in fraud detection. And I believe the community would agree with me on this.
", "votes": 3, "replies": [] }, { "id": 2917052, "author_name": "kurupical", "author_url": "", "post_date": "2024-07-11T13:01:41.020000", "content": "I consider the current situation to be \"unwanted but not prohibited\" and in that sense I understand that the situation has not changed. (but this is my opinion and I'm sure there are many different arguments) From your past statements, I mistakenly thought that you had the same opinion as mine. If you consider the current situiation to be \"unwanted and prohibited\" and think from your point of view, your statement is right.
\nKaggle very simple, really. EVERYTHING allowed… except what written Rules .\nDoes anything said , 'multi-row--single-target approach forbidden'?\nNo. Hence, allowed.\nyes but the host declared clearly that multi-to-single are not allowed. This might not be writen but it is obivioustly a new rule, no?
", "votes": 0, "replies": [] }, { "id": 2917067, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T13:08:10.607000", "content": "I agree with you that it is \"unwanted but not prohibited\". Since it is unwanted, I prefer to change the situation to be 'wanted'. Using leak is OK but it's a risk if leak is leaked before the end of the competition. This is also part of the game IMO.
", "votes": 1, "replies": [] }, { "id": 2917075, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T13:17:55.463000", "content": "@vasileioscharatsidis Yes, I agree with you.
", "votes": 1, "replies": [] }, { "id": 2917080, "author_name": "kurupical", "author_url": "", "post_date": "2024-07-11T13:23:42.517000", "content": "\n\nSince it is unwanted, I prefer to change the situation to be 'wanted'.
\n
I understand 'wanted' -> 'host wanted'.
\nIf I was a host and be forced to allowed to use leak, I don't want to extent the deadline. Because whether extenting or not, there will be only solution using leak. (But 'some participants wanted' to extent in allowed case.)
\n\nUsing leak is OK but it's a risk if leak is leaked before the end of the competition. This is also part of the game IMO.
\n
I agree!
", "votes": -2, "replies": [] }, { "id": 2917091, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T13:28:31.827000", "content": "\n\nIf I was a host and be forced to allowed to use leak, I don't want to extent the deadline. Because whether extenting or not, there will be only solution using leak. (But 'some participants wanted' to extent in allowed case.)
\n
Ultimately what host want is the most important.
\nIf host want to finish in time, so be it.
\nBut since you shared why YOU don't want an extension, I will share the other side too.
@jerrylin96 If there exists even one participant who intends to stick with not using the leak and has the balls to try to be competitive even against leak-based approaches, then I feel we should give an extension, even if just for a week.
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519249#2915873
", "votes": 0, "replies": [] }, { "id": 2917107, "author_name": "kurupical", "author_url": "", "post_date": "2024-07-11T13:34:16.850000", "content": "Of course, if leaks are allowed and there are no extensions, that would be great for me.
\nBut I'm not making this statement because I want to win, I'm just really tired and want it to be over quickly😂
\n\nIf there exists even one participant who intends to stick with not using the leak and has the balls to try to be competitive even against leak-based approaches
\n
Look at the difference between me and current 1st place. And believe me, I did a LOT.
\nThis is how 'competitive' you can be against leak in the high ranks.
\nHowever, extension is good. Since then, I can try to bring a leaky solution too :)
Thank you for sharing your thought.
\nI thought you use leakage until yesterday, so I'm very surprised to know your achievement is without leakage.
Whatever decision the host makes, let's do our best (of course, within the rules!).
\nI'm looking forward to read your solution.
@shlomoron
\nYou are so energetic🤣 To be honest, I have no power or interest in utilizing leaked information. I will give up if the host decides to allow leaky solutions. (But I am betting that the host will only disqualify explicit multi-to-single column predictions at test time.)
\n\nI thought you use leakage until yesterday
\n
So you guys tried to use leak because you can't catch up?
", "votes": 1, "replies": [] }, { "id": 2917138, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T13:58:21.190000", "content": "\n\nI thought you use leakage until yesterday, so I'm very surprised to know your achievement is without leakage.
\n
Why you thought so? I'm only ~0.001 above your solution. It's within reason. Yep, my solution is leak-free and is truly a great one if I say so about myself lol. (Also, it will stay a secret if I don't finish in price range 🤣) Anyway, thank you.
", "votes": 5, "replies": [] }, { "id": 2917143, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T14:02:47.983000", "content": "\n\nYou are so energetic🤣 To be honest, I have no power or interest in utilizing leaked information. I will give up if the host decides to allow leaky solutions
\n
Hahaha I had several great insights and I'm pretty sure they can be carried on at least partly to leaky solutions. Even If I don't succeed to develop one myself, I only need to find a team with working leaky solution and I would probably be able to contribute much. So yeah I might be tired, but I won't go down without a fight lol
", "votes": 0, "replies": [] }, { "id": 2917151, "author_name": "Federico Peccia", "author_url": "", "post_date": "2024-07-11T14:07:18.197000", "content": "This is the third time I read a comment from you talking about your great solution. I am really rooting for you, I want to know what the hell you discovered! hahaha
", "votes": 1, "replies": [] }, { "id": 2917153, "author_name": "heng", "author_url": "", "post_date": "2024-07-11T14:07:44.850000", "content": "Respect! You deserve the championship and the prize.
", "votes": 1, "replies": [] }, { "id": 2917158, "author_name": "Truth Seeker", "author_url": "", "post_date": "2024-07-11T14:08:21.237000", "content": "Once upon a time, when \"kernel trick\" still reigned, no one cared about neural networks. Shunning away from NN was so prevalent that if your paper was about NN, your chance of publication would be very low.
\nIn 2012, a dude by the name of Alex Krizhevsky had the balls to use NN and competed, and the rest was history.
\nMoral of the story: Never underestimate someone who has balls.
\nMoral 2 of the story: A small extension to the deadline is a very small price to pay for a chance of a good solution @jerrylin96 intends.
if leaks are allowed and there are no extensions, that would be great.
Are you serious? hahaha. 4 days left.
\nWhy not simply have the person who discovered the leak fill out their home address? We will send the prize and gold medal to you via express delivery.
", "votes": 0, "replies": [] }, { "id": 2917176, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T14:17:03.307000", "content": "\n\nWhy not simply have the person who discovered the leak fill out their home address? We will send the prize and gold medal to you via express delivery.
\n
Well, since his team already have a prepared leaky solution from previous leak…lol
", "votes": 2, "replies": [] }, { "id": 2917182, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T14:23:00.123000", "content": "I think so. :) 🥺.
\nNow, I realize that those strange accounts who leak the leakage in the discussion might be friendly to us. They brought leakage into public and start a huge debate. Otherwise, leaky team will go to the top in the last second.
\nThank you. @rkwasny @bethegoose
", "votes": 1, "replies": [] }, { "id": 2917191, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T14:28:16.363000", "content": "\n\nNow, I realize that those strange accounts who leak the leakage in the discussion might be friendly to us. They brought leakage into public. Otherwise, leaky team will go to the top in the last second.
\n
I don't know if they had good intentions, but it was definitely good that they revealed the leak. Imagine the horror of reaching the last day and suddenly finding five leaky teams on top. I shudder to think about it.
", "votes": 3, "replies": [] }, { "id": 2917199, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-11T14:36:58.570000", "content": "I don't know if they had good intentions
Yeah. I have a hunch that it's a team that already leveraged a leak to score impressively offline, fearing disqualification at the end. Thus, they chose an awkward moment to expose the leak, assuming others wouldn't have time to use the leak to outperform them.
\nAfter the competition closed, they could justify their actions by citing that \"'others' had already published the leak\". And they are very likely to get any penalties.
\nThey just didn't expect such a strong community reaction….
", "votes": 0, "replies": [] }, { "id": 2917201, "author_name": "heng", "author_url": "", "post_date": "2024-07-11T14:38:56.767000", "content": "\n\nImagine the horror of reaching the last day and suddenly finding five leaky teams on top. I shudder to think about it.
\n
This has happened in foursqure competition. Some participants are very good at digging out leaks and hiding it until the last minute.
", "votes": 2, "replies": [] }, { "id": 2917215, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T14:48:19.750000", "content": "\n\nThis has happened in foursqure competition. Some participants are very good at digging out leaks and hiding it until the last minute.
\n
Yeah…sadly I'm not very good at finding leaks, only at building models :(
", "votes": 0, "replies": [] }, { "id": 2917225, "author_name": "heng", "author_url": "", "post_date": "2024-07-11T14:54:00.857000", "content": "Hahaha, me neither. I just want to finish this competition now and start playing the Elden Ring DLC!
", "votes": 1, "replies": [] }, { "id": 2917236, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T14:56:42.097000", "content": "\n\nHahaha, me too. I just want to finish this competition now and start playing the Elden Ring DLC!
\n
Forget Elden Ring…you would not find a greater drama than Kaggle drama 🤣
", "votes": 0, "replies": [] }, { "id": 2917507, "author_name": "Psi", "author_url": "", "post_date": "2024-07-11T16:51:08.647000", "content": "extending competition is IMO only a last resort, specifically close to deadlines
\nkaggle requires lots of dedication and people plan their work and vacation around it
I hope the competition can finish on time. Although my ranking is not high, I have made many attempts and I'm a bit tired.
", "votes": 2, "replies": [] } ] }, { "id": 2916481, "author_name": "Camaro", "author_url": "", "post_date": "2024-07-11T04:05:08.313000", "content": "@jerrylin96
\nHere is (relatively) simple, clear and feasible solution for the leak prevention.
\n\nGold medal team must re-submit their solution via time-series api after finalization(no extension).
\n
Agree. But I think there is a possibility to store the data and then process it for inference. And this requires Kaggle to redesign a submission pipeline, which is a bit troublesome for them.
", "votes": 1, "replies": [] } ] }, { "id": 2919526, "author_name": "Akansha_Bhardwaj1818", "author_url": "", "post_date": "2024-07-13T01:43:54.017000", "content": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it. Maybe i am too romantic or too new into this …
", "votes": 6, "replies": [ { "id": 2920030, "author_name": "steubk", "author_url": "", "post_date": "2024-07-13T11:39:12.023000", "content": "There is a long tradition of leaks in Kaggle competitions, and in many cases, they have been made public as soon as they were discovered.
\nSome historical examples that come to mind:
\nAs Kagglers, we have repeatedly urged for a solution that includes a (honor) reward for discoverers, which would serve as an incentive to disclose:
\nIn any case, here you can find a video with an interesting perspective by a GrandMaster
\nhttps://www.youtube.com/watch?v=rrWmmqAt6Ts
@jerrylin96
\nI can come up with following choice.
Now I prefer option 1 😭
\nAre there other columns that could cause leaks?
", "votes": 0, "replies": [] }, { "id": 2916524, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-11T04:38:29.190000", "content": "For 1 we need to remove also pbuf_SOLIN. And maybe additional columns. And we STILL have the risk of yet another similar leak down the line.
\nFor 3, I hope that it come with an extension so I can utilize leak too…
I preferred 2. Tired……
", "votes": 2, "replies": [] }, { "id": 2916579, "author_name": "Bilzard", "author_url": "", "post_date": "2024-07-11T05:47:03.377000", "content": "I disagree option 1. This leak is probably intrinsic in the ClimSim dataset. We can't know in advance which feature contains location/timestamp information. I think if host fixed one leak, then someone will find out another leak. It will be endless.
", "votes": 3, "replies": [] }, { "id": 2916593, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-11T06:00:27.240000", "content": "Coding check need to cover train part. Only checking inference inputs is not enough.
\nOtherwise, kagglers can use 2d -> 1d models to get oof and then use pseudo labels or distillation to learn this oof.
", "votes": 4, "replies": [] }, { "id": 2916598, "author_name": "Andrew Geiss", "author_url": "", "post_date": "2024-07-11T06:04:34.687000", "content": "Both of these variables are important for radiative transfer. Ozone has an impact on stratospheric heating rates in particular and insolation is important throughout the column. From a climate modeling perspective you cannot remove them.
", "votes": 5, "replies": [] }, { "id": 2916763, "author_name": "hyd", "author_url": "", "post_date": "2024-07-11T08:34:28.657000", "content": "For choice 1, the new dataset should be with irregular timestamp gap(which is constant now), better with a time-series API and another 15 days extension.
\nFor choice 2, it's terrible. It hurts kaggle and host's reputation. It reminds me of bad memories of comps at Chinese Tianchi, I was always thinking some teams cheated but got away with being punished.
\nFor choice 3, another 15 days extension also needed(maybe not enough).
\n\nFor choice 2, it's terrible. It hurts kaggle and host's reputation. It reminds me of some bad memories of comps at Chinese Tianchi, I was always thinking some teams cheated but got away with being punished.
\n
You are right, even though I am the OP who originally requested the host to disqualify. Even if they do it, we will forever suspect that some winners have hidden a cheating method behind their code.
\nSo it will probably be best to extend comp and allow leak.
\nI will be bold and say that even one month. Since with leak it turn to almost a new competition.
\nThe other choice is for the host to produce an extremely irregular and downsampled test, both in time and space (i.e., something like ~50/384 points per timestep). I don't know how much computation it will require. Even then, there is a risk of finding some extraordinary leak.
\nUnfortunately, time series API is useless in this competition since, unlike time series, we want inference to be independent not only of the latter samples but also of the previous ones. For example, with time series API, a user can save the first samples and then slowly build a 'picture' of complete grids and turn it to 2D->1D for the latter samples.
\nSo, all in all, it's probably best to just give us time-space data and turn it into a 2D->1D competition with a massive (as I said, maybe even ~1 month) extension.
\nHonestly, this is what the host should have done from the beginning. As someone said, 'Never underestimate Kagglers.' The more someone tries to bind us to a particular method, the more loopholes we will find. Leading to 'leak competition' instead of 'DL competition' which just ruin the competition for everyone except the few that found leak.
What about timeseries api with the requirement that the inference code be made public/available - much easier to check for irregularities if just dealing with the timeseries api?
", "votes": 0, "replies": [] }, { "id": 2916791, "author_name": "heng", "author_url": "", "post_date": "2024-07-11T08:56:57.183000", "content": "It would be better to extend the deadline for team-up by one week.
\nI don’t want to go solo anymore.
\nI'm really tired. 😂
\n\nI don’t want to go solo anymore.
\n
Nooo I was rooting for you to become a GM :(
\nBut yeah if it turn to 2D->1D comp' with extension I don't know if I will have the energy to remain solo lol
\n\nWhat about timeseries api with the requirement that the inference code be made public/available - much easier to check for irregularities if just dealing with the timeseries api?
\n
'Never underestimate Kagglers'- SOMEONE will probably find a loophole or succeed to hide his cheating 🤣
", "votes": 2, "replies": [] }, { "id": 2916808, "author_name": "", "author_url": "", "post_date": "2024-07-11T09:06:41.637000", "content": "", "votes": 0, "replies": [] }, { "id": 2916809, "author_name": "", "author_url": "", "post_date": "2024-07-11T09:06:58.063000", "content": "", "votes": 0, "replies": [] }, { "id": 2916810, "author_name": "heng", "author_url": "", "post_date": "2024-07-11T09:09:47.823000", "content": "This goes along with the advice I gave here earlier: 😀
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519249#2916364
\nBut I think coding check should need to cover train part. as @hookman said.
All top teams now have good reputation, only few teams would take that risk.
\nalso concat with irregular historical data can not boost much score and is time-consuming both in training and inference phase.
\nalso inference code is much easier to check for us than rerun all their training code.
\nso i think this choice is fine!
\nhidden test and time series API are needed.
\n\nAll top teams now have good reputation, only few teams would take that risk.
\n
'All top teams now have good reputation,'
\nFor now. But with leak you cannot know how many with less reputation will jump to top 10…
\nBut maybe the situation is still salvageable indeed with irregular test+time-series API+public inference code. You are making strong points.
I think 2) is not possible as many will use the leak implicitly. It has happened many times before that eg larger and more complex models were beneficial for leaky data as they can capture the leak better.
", "votes": 4, "replies": [] } ] }, { "id": 2915373, "author_name": "Bilzard", "author_url": "", "post_date": "2024-07-10T13:21:09.903000", "content": "@shlomoron Well, this discussion's opinion seems contradict to your previous comment. Have you change your mind?
\n\n\nKaggle is very simple, really. EVERYTHING is allowed…except what is written in the Rules tab.
\n
\n Does anything in said tab say, 'multi-row-to-single-target approach is forbidden'?
\n No. Hence, it is allowed.
https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/514626#2888661
", "votes": 6, "replies": [ { "id": 2915376, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-10T13:24:40.863000", "content": "If this leak was not revealed before competition end, I would not be able to say anything. Since it is allowed by the rules.
\nSince it was found before the end of the competition, I can ask for the host to 'soildify' their 'spirit of the competition' in the rules, as it became relevant again.
\nNo contradictions here imho.
yes but in the meantime the hosts made clear that multi-row-to-single-target is not allowed.
", "votes": 1, "replies": [] }, { "id": 2915479, "author_name": "Truth Seeker", "author_url": "", "post_date": "2024-07-10T14:29:59.760000", "content": "This discussion thread of @shlomoron (known as greySnow) does not contradict his/her earlier comment when greySnow said \"EVERYTHING is allowed…except what is written in the Rules tab.\"
\nIt is one thing how one interprets the exact definition of Rules with respect to what're technically allowed. It is entirely another thing whether one decides to do it or not (based possibly on moral or other grounds). greySnow and I may had some contention about the former, but certainly not the latter.
", "votes": 0, "replies": [ { "id": 2915503, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-10T14:42:38.703000", "content": "\n\ngreySnow and I may had some contention about the former, but certainly not the latter.
\n
Thank you for your support, but we actually are also contested on the latter 😅
\nI think it is entirely moral to utilize leak…
\nBut since the leak was revealed, I can ask for better rules since it's in my best interest. (Well, and in the best interest of most participants, though most certainly not ALL.)
\nThis way or another, it's up to the host. Nothing moral or immoral either way.
\nAnd my techniques will lend me gold, if not in this competition, then in the next. So it's not the end of the world even if leak is used…
I knew this was coming. That's why I added \"moral or other grounds\" 😅
\nAnd that \"other grounds\" could certainly be one's best interest. Not a problem at all.
\nSo, I'm certainly correct in saying that we're not contested in the latter.
\nAnd this post of yours certainly has zero contradiction to your earlier comment.
", "votes": 1, "replies": [] }, { "id": 2915544, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-10T14:55:43.020000", "content": "Hahaha you got me there
", "votes": 1, "replies": [] } ] } ] }, { "id": 2915991, "author_name": "Sarah Jeffreson", "author_url": "", "post_date": "2024-07-10T18:03:42.240000", "content": "Just two cents from a Kaggle newbie who has little skin in the game time-investment-wise: it seems to me (but feel free to argue!) that solutions utilizing this leak will be useless to the host/climate community. As far as I can see, the reason that pbuf_ozone gives the location/time-stamp is just because these values have been fixed to constants in the individual E3SM-MMF simulations. It's not because ozone is truly a predictor of location/time in the real world. The ClimSim paper explicitly states that this \"limitation\" will be improved in subsequent models. Once it has been fixed, any model that assumes all data-points with the same pbuf_ozone are related, and uses many to predict one, may not just lose their advantage, they may perform significantly worse than the one-to-one solutions.
\nTherefore, if the prizes are supposed to go to those who produce something useful for the host (which seems reasonable to me), it would not be unfair to disqualify the leak-based solutions.
", "votes": 3, "replies": [] }, { "id": 2915682, "author_name": "Yusef A.", "author_url": "", "post_date": "2024-07-10T15:54:17.820000", "content": "Not sure a disqualification is a fair response given it's so close to the end of the comp - seems unfair for those who found it independently and managed to make use of it…
\nFWIW - we aren't using it.
", "votes": 1, "replies": [ { "id": 2915693, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-10T15:58:56.050000", "content": "It's unfair either way. If host does not disqualifiy, it's unfair for all teams the invested a lot of effort in good faith after host stressed that this would be 1D->1D and even changed test set and extended the competition for this…but fairness is just a word. The world is unfair and it's ok lol.
", "votes": 4, "replies": [] } ] }, { "id": 2915093, "author_name": "heng", "author_url": "", "post_date": "2024-07-10T10:27:21.647000", "content": "Completely agree.
\nBut this means that organizers need additional energy and technology to detect these leak-based solutions, and there may be some omissions.
", "votes": 1, "replies": [ { "id": 2915111, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-10T10:38:56.493000", "content": "True, it would require energy and effort. I can only request.
", "votes": 1, "replies": [] } ] }, { "id": 2915679, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-07-10T15:53:00.243000", "content": "I understand that being #1 on public LB, this isn't good news, but that is the nature of Kaggle competitions. It is the Optiver competition from 2021 all over again:
\nhttps://www.kaggle.com/competitions/optiver-realized-volatility-prediction/discussion/284121
It was a very similar situation, except that only 10-20 teams used the leak and it wasn't made public. Admins made changes to the data to effectively disqualify these teams. However, this created a lot of backlash in the community and they ultimately decided to revert the change, allowing the leak (the leak was much, much worse/ gave a massive boost). Now there is a small difference in that the leak was made public a few days before the end of the competition, but disqualifying leak solutions would also make it unfair for teams who assumed everything is fair game and have used the leak all the way. Either way, I think it isn't great and an extension of the timeline would be the least bad thing the hosts could do in my opinion (if they can afford that).
", "votes": 2, "replies": [ { "id": 2915741, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-10T16:14:28.603000", "content": "\n\nunfair for teams who assumed everything is fair game and have used the leak all the way
\n
After the hosts made his point time and again about the 'spirit of the competition', including changing the test set once and extending the competition (without any backlash, mind you), I would say that it's hardly unfair. More like 'professional risk' that leak utilizers took knowingly…but it's up to the host. And yes, if he does not disqualify, I at least hope for an extension to be able to utilize the leak properly, too. Even tho I'm tired, lol
", "votes": 2, "replies": [ { "id": 2915789, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-07-10T16:38:33.017000", "content": "As Jerry Lin said:
\n\n\nKnowing this, it is fairly obvious that the intention of the competition designers was to not grant you access to this information. As Kaggle user @ryches puts it, using this information indeed goes against the spirit of the competition.
\n
This doesn't sound like a rule which is enforced upon the participants to me. Similar things were said in the Optiver competition when there was suspected there might be a leak.
\nBut in the end, the Kaggler is in a tough spot; utilize the leak if they found it (since why not, every other winning team uses some kind of leak in their solution, and how else to be competitive if other top teams use it), or not use it, possibly ending in a bad place on the LB even though the solution was great. Or be so courageous to make the leak public (and I don't mean a few days before end). Anyway, given how widespread and frequent the utilization of leaks is in kaggle competitions (and I think it is clear that this is almost always against the hosts intention and not in the spirit of the competition), it would be a strange exception to make.
\nI think it would be fine if Kaggle would clearly state in the beginning of a competition, that such solutions will be disqualified, but AFAIK this wasn't the case here. Also it is hardly detectable for non-prize winning solutions.
\nI also only heard about this leak a few hours ago. But of course it makes you think, if top teams are now required to get a solution out there which makes use of the leak in a final sprint ?
I agree that leak utilization is OK. Ultimately, it's not a question of fairness but what is best for the host. I can only make my request in my (obviously) best interest. Since rule change by last 5 days is withing the rules, and align with host's intention, I don't think there is a problem with that, even though it's 'unfair for those who utilized leak' (knowing that there is a chance that leak is leaked before comp' end leading to their efforts wasted).
", "votes": 0, "replies": [] }, { "id": 2915873, "author_name": "Truth Seeker", "author_url": "", "post_date": "2024-07-10T17:20:42.370000", "content": "For the record, let me say this: I will stick with not using any leak and genuinely treat this as an i.i.d. regression problem. All I ask of @jerrylin96 is one more week (for a deadline extension to Jul 22) so I can use every last drop of my brain juice to try to remain competitive even against approaches using the leak.
\nConsider this an official contract signed with my soul.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2915077, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-07-10T10:17:37.687000", "content": "Totally agree with you and can't wait for reading your solution
", "votes": 2, "replies": [] }, { "id": 2915110, "author_name": "hyd", "author_url": "", "post_date": "2024-07-10T10:38:55.167000", "content": "Totally agree!
\nI suggest top 50 teams to make a promise here that your team will not use this trick.😀
Hahaha for silver range there is probably not much to do. It's too many solutions to keep track on.
\nBut for gold range (and surly for price) it's possible.
nice tutorial
", "votes": -2, "replies": [] }, { "id": 2920981, "author_name": "Fernando Melo", "author_url": "", "post_date": "2024-07-14T01:17:19.867000", "content": "What do you mean by 2D > 1D? What does 2D mean?
", "votes": 0, "replies": [] }, { "id": 2920108, "author_name": "uthejitha Moore", "author_url": "", "post_date": "2024-07-13T12:37:27.537000", "content": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it
", "votes": 0, "replies": [] }, { "id": 2919221, "author_name": "Ahmed Arham", "author_url": "", "post_date": "2024-07-12T18:37:06.683000", "content": "Is the timestamp/district for new test set perfectly recovered?
", "votes": 0, "replies": [] }, { "id": 2918240, "author_name": "S Jyothirmayee", "author_url": "", "post_date": "2024-07-12T05:31:49.660000", "content": "hello good outcome
", "votes": 0, "replies": [] }, { "id": 2916434, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-07-11T03:00:09.920000", "content": "Is the timestamp/district for new test set perfectly recovered? 🤔
", "votes": 0, "replies": [] }, { "id": 2916250, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-10T20:47:24.640000", "content": "Thanks for the suggestion! We are looking into this.
", "votes": 0, "replies": [] }, { "id": 2916436, "author_name": "", "author_url": "", "post_date": "2024-07-11T03:02:38.087000", "content": "", "votes": -1, "replies": [] }, { "id": 2921801, "author_name": "VanillB", "author_url": "", "post_date": "2024-07-14T16:56:15.587000", "content": "Thanks for the nice information
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2915072": "It turns out that [due to pbuf_ozone_2 leak](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519184), test set timestamp and location can be exactly recovered. This leak enables 2D->1D models and possibly also future data utilization (using the input for future samples to 'predict' the output of a previous one).\nI humbly request the hosts disqualify all techniques that explicitly utilize this (and possibly other similar) leak. It was repeatedly stressed by the hosts that this competition is to be a 1D->1D regression problem, and I, and I think this is also true for many others, invested extensive time and effort (and possibly money for compute) based on this assumption. All of this will become meaningless now if teams start to utilize the leak.\nOn a personal note, I made a lot of interesting things and utilized advanced techniques in this competition, which I would keep secret for future competitions if I drop from the price range due to a leak, as I'm sure that those techniques would easily lend me gold in future relevant competitions.", "2917435": "From Section B.10 :\n\n\"Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.\n\nA disqualified participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.\"\n\nWe will disqualify leak-based solutions for cash prizes. To be eligible for a cash prize, Kaggle teams will have to verify their training and inference code with the competition hosts.", "2916872": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it. Maybe i am too romantic or too new into this ...", "2916314": "Hello all. I am waiting to confirm this with the Kaggle staff, but it may be safe to assume for now that we will disqualify any submissions that take advantage of any leak or use a multicolumn approach for cash prizes and gold medals.", "2915289": "**New Competition Idea: Predict the next deadline extension**", "2916993": "@jerrylin96 \nWhatever you decide, I hope you will not extend the competition. You previously referred to the 2 week extension as \"good news\", but it is clearly \"bad news\" as evidenced by participents reactions. \nWe kagglers are more committed to kaggle than you might think. We are working as hard as we can towards the end of the competition period. We have a schedule that is tailored to the competition. \nIn the past, when the competition was extended, some people said \"I was planning a trip right after the competition was over\"(*1) \n\nExtending this further would be like running a full marathon, reaching the 40 km mark, and starting the final sprint, only to announce that the finish line has been changed to 50 km.\n\n(*1) https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/110777#637593", "2916481": "@jerrylin96 \nHere is (relatively) simple, clear and feasible solution for the leak prevention.\n- Gold medal team must re-submit their solution via time-series api after finalization(no extension).\n- At training time, allow everything. Detecting use of leakage at training time is very costful or sometimes impossible.", "2919526": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it. Maybe i am too romantic or too new into this …", "2916506": "@jerrylin96 \nI can come up with following choice.\n1. Refresh **test.csv** again and remove **pbuf_ozone**, and then extend deadline.\n2. Not allow the use of leak. Coding checks and reproducing results are mandatory(at least for gold zone).\n3. Allow the use of leak.", "2915373": "@shlomoron Well, this discussion's opinion seems contradict to your previous comment. Have you change your mind?\n\n> Kaggle is very simple, really. EVERYTHING is allowed…except what is written in the Rules tab.\n> Does anything in said tab say, 'multi-row-to-single-target approach is forbidden'?\n> No. Hence, it is allowed.\n\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/514626#2888661", "2915991": "Just two cents from a Kaggle newbie who has little skin in the game time-investment-wise: it seems to me (but feel free to argue!) that solutions utilizing this leak will be useless to the host/climate community. As far as I can see, the reason that pbuf_ozone gives the location/time-stamp is just because these values have been fixed to constants in the individual E3SM-MMF simulations. It's not because ozone is truly a predictor of location/time in the real world. The ClimSim paper explicitly states that this \"limitation\" will be improved in subsequent models. Once it has been fixed, any model that assumes all data-points with the same pbuf_ozone are related, and uses many to predict one, may not just lose their advantage, they may perform significantly worse than the one-to-one solutions.\n\nTherefore, if the prizes are supposed to go to those who produce something useful for the host (which seems reasonable to me), it would not be unfair to disqualify the leak-based solutions.", "2915682": "Not sure a disqualification is a fair response given it's so close to the end of the comp - seems unfair for those who found it independently and managed to make use of it...\n\nFWIW - we aren't using it.", "2915093": "Completely agree. \n\nBut this means that organizers need additional energy and technology to detect these leak-based solutions, and there may be some omissions.", "2915679": "I understand that being #1 on public LB, this isn't good news, but that is the nature of Kaggle competitions. It is the Optiver competition from 2021 all over again:\nhttps://www.kaggle.com/competitions/optiver-realized-volatility-prediction/discussion/284121\n\nIt was a very similar situation, except that only 10-20 teams used the leak and it wasn't made public. Admins made changes to the data to effectively disqualify these teams. However, this created a lot of backlash in the community and they ultimately decided to revert the change, allowing the leak (the leak was much, much worse/ gave a massive boost). Now there is a small difference in that the leak was made public a few days before the end of the competition, but disqualifying leak solutions would also make it unfair for teams who assumed everything is fair game and have used the leak all the way. Either way, I think it isn't great and an extension of the timeline would be the least bad thing the hosts could do in my opinion (if they can afford that).", "2915077": "Totally agree with you and can't wait for reading your solution", "2915110": "Totally agree!\nI suggest top 50 teams to make a promise here that your team will not use this trick.😀", "2916625": "nice tutorial", "2920981": "What do you mean by 2D > 1D? What does 2D mean?", "2920108": "What is very dissapointing as a new kaggler is that i feel like i am the only one or at least part of a minority thinking that if a leak is found the moral obligation is to immidiatelly make it known publicly and definately not use it", "2919221": "Is the timestamp/district for new test set perfectly recovered? \n\n", "2918240": "hello good outcome", "2916434": "Is the timestamp/district for new test set perfectly recovered? 🤔\n", "2916250": "Thanks for the suggestion! We are looking into this.", "2916436": "", "2921801": "Thanks for the nice information" } }