On the private leaderboard of this competition there are at least 18 teams with all the typical and many times described signs of involvement in the solutions (medals) sell-buy business. They are visibly clustered between places 43 and 65.
\nRecently from all leaderboards of competitions without a significant shake up and from various public comments and testimonies it could be judged that the medals sell-buy business has been flourishing.
\nSeveral times Kaggle team got a lot of well documented detailed information about how the business works from decent Kagglers (including Grandmasters) who speak the language. The links to the business websites were also published.
\nAll the competitions are code competitions so it should be easy to compare the codes of submissions of visibly suspicious teams. Does Kaggle team routinely do it?
I cannot say whether Kaggle team reacted anyhow internally to all the given evidence but the results were invisible and the leaderboards stayed nearly as they had been.
\nAll this necessarily leads to the question:
\n@wcukierski, @addisonhoward, @maggiemd in your opinion and according to Kaggle rules is buying a solution a newly authorized way how to participate in a Kaggle competition and get all possible awards?
On the private leaderboard of this competition there are at least 18 teams with all the typical and many times described signs of involvement in the solutions (medals) sell-buy business. They are visibly clustered between places 43 and 65.
\nRecently from all leaderboards of competitions without a significant shake up and from various public comments and testimonies it could be judged that the medals sell-buy business has been flourishing.
\nSeveral times Kaggle team got a lot of well documented detailed information about how the business works from decent Kagglers (including Grandmasters) who speak the language. The links to the business websites were also published.
\nAll the competitions are code competitions so it should be easy to compare the codes of submissions of visibly suspicious teams. Does Kaggle team routinely do it?
I cannot say whether Kaggle team reacted anyhow internally to all the given evidence but the results were invisible and the leaderboards stayed nearly as they had been.
\nAll this necessarily leads to the question:
\n@wcukierski, @addisonhoward, @maggiemd in your opinion and according to Kaggle rules is buying a solution a newly authorized way how to participate in a Kaggle competition and get all possible awards?
Now it's purging time, lets wait until the LB finalize and see if the suspected teams are removed…
", "rawMarkdown": "Now it's purging time, lets wait until the LB finalize and see if the suspected teams are removed...", "votes": 6 }, { "id": 3034171, "postDate": "2024-11-01T21:11:15.037Z", "content": "It really looks odd, especially since some of them participated in other competitions and also landed in a similar cluster… But it seems they haven't been removed. E.g. the RSNA competition, ranks 40-50:
\nhttps://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/leaderboard
\n@sohier was this actually investigated? I trust the kaggle team though, if they say this is all valid
As you mentioned the last RSNA competition - a similar discussion was led there too.
\nAnd both competitions were finalized too quickly (in less than 20 hours) to believe that any further investigation had been done.
I really don't understand the reason but the current Kaggle competitions administration seems to lack any will to crack down on this problem.
\nUnfortunately this situation devalues all medals and awards.
Maybe kaggle does not care about silver/bronze medals value since I see that the suspected teams are still there, and they are definitely suspicious.
", "rawMarkdown": "Maybe kaggle does not care about silver/bronze medals value since I see that the suspected teams are still there, and they are definitely suspicious." }, { "id": 3034465, "postDate": "2024-11-02T07:04:49.483Z", "content": "That would be really sad. For many participants, winning bronze or silver medals is a great achievement and they invest many weeks of their free time in the competitions. This can't be right from the company's point of view either, because it could deter participants from taking part in Kaggle competitions in the future.
\nAny form of private sharing, cheating or purchased solutions should be excluded as far as possible (maybe that would be an interesting competition: predicting whether a solution was legit or not 😃 ). Personally, I'm fine with validation taking a few days (as in the past)…
", "rawMarkdown": "That would be really sad. For many participants, winning bronze or silver medals is a great achievement and they invest many weeks of their free time in the competitions. This can't be right from the company's point of view either, because it could deter participants from taking part in Kaggle competitions in the future.\n\nAny form of private sharing, cheating or purchased solutions should be excluded as far as possible (maybe that would be an interesting competition: predicting whether a solution was legit or not 😃 ). Personally, I'm fine with validation taking a few days (as in the past)...", "votes": 7 } ] } ] }, { "id": 3036791, "postDate": "2024-11-04T23:22:20.143Z", "content": "We would like to provide some insight into how we maintain fairness and integrity within our competitions. We screen competitions after the close (if done during the competition, teams could re-enter under new accounts), and because we do not publicly disclose our actions, there is sometimes a misconception that Kaggle ignores evidence of foul play. Kaggle does not overlook evidence of rules violations, nor give unequal treatment based on leaderboard rank, account tenure, progression status, or any other extenuating factor.
\nBelow is an overview of the steps we take to identify and address cheating and guidance on reporting any concerns.
\nBehind the scenes, our team monitors various activities, including leaderboards, submissions, user behavior, and team formation/merging. While we do not publicly address user-reported evidence of cheating, we are engaged with our community members who report suspicious activity. Details on how to report cheating are found below.
\nOnce a competition ends, our compliance team conducts an in-depth review, combining automated checks with manual oversight. Though we do not reveal the details of our review process, we listen to and have implemented community-sourced ideas to strengthen this process.
\nWe added an account verification process in early 2024 to deter cheating up front. Since adding this requirement, we have observed a marked drop in attempts to game competitions. Please see our announcement for more detail on the verification process.
\nWe encourage community members to report any suspicious activity or rule violations. Our compliance team evaluates each and every report.
\nTo report a concern, please:
\nSubmit a Ticket: Visit Support/Contact and navigate to \"I’d like to report cheating in a Competition.\" If you believe you have sufficient evidence that warrants an investigation, please proceed with submitting a ticket. We manually review every report received.
Provide Specific Details: Include team names, usernames, observed behaviors, screenshots, and, if available, screenshots or other supporting evidence.
All reports are handled confidentially. While we may not share the specific outcomes of each report, we value community input and use these to improve our effectiveness.
\nThe strength of evidence matters. It is not sufficient for Kaggle to take action on speculation or the mere appearance of wrongdoing. We must corroborate reports with concrete evidence of rules violations.
\nWe understand that some teams may be flagged inadvertently. While we’ve removed thousands of teams from competitions, mistakes happen. If you believe your team was falsely identified and removed, we have have a simple appeals process:
\nWe hope this provides some clarification on the actions we take to maintain fair play within our competitions.
", "rawMarkdown": "We would like to provide some insight into how we maintain fairness and integrity within our competitions. We screen competitions after the close (if done during the competition, teams could re-enter under new accounts), and because we do not publicly disclose our actions, there is sometimes a misconception that Kaggle ignores evidence of foul play. Kaggle does not overlook evidence of rules violations, nor give unequal treatment based on leaderboard rank, account tenure, progression status, or any other extenuating factor.\n\nBelow is an overview of the steps we take to identify and address cheating and guidance on reporting any concerns.\n\n# Active Monitoring\nBehind the scenes, our team monitors various activities, including leaderboards, submissions, user behavior, and team formation/merging. While we do not publicly address user-reported evidence of cheating, we are engaged with our community members who report suspicious activity. Details on how to report cheating are found below.\n\n# Post-Competition Screen \nOnce a competition ends, our compliance team conducts an in-depth review, combining automated checks with manual oversight. Though we do not reveal the details of our review process, we listen to and have implemented community-sourced ideas to strengthen this process.\n\n# Verification\nWe added an account verification process in early 2024 to deter cheating up front. Since adding this requirement, we have observed a marked drop in attempts to game competitions. Please see our [announcement](https://www.kaggle.com/discussions/general/490841) for more detail on the verification process.\n\n# Reporting Cheating\n**We encourage community members to report any suspicious activity or rule violations.** Our compliance team evaluates each and every report.\n\nTo report a concern, please:\n\n- **Submit a Ticket:** Visit [Support/Contact](https://www.kaggle.com/contact) and navigate to \"I’d like to report cheating in a Competition.\" If you believe you have sufficient evidence that warrants an investigation, please proceed with submitting a ticket. We manually review every report received. \n\n- **Provide Specific Details:** Include team names, usernames, observed behaviors, screenshots, and, if available, screenshots or other supporting evidence.\n\nAll reports are handled confidentially. While we may not share the specific outcomes of each report, we value community input and use these to improve our effectiveness.\n\n**The strength of evidence matters.** It is not sufficient for Kaggle to take action on speculation or the mere appearance of wrongdoing. We must corroborate reports with concrete evidence of rules violations.\n\n# Appeals and Safeguards Against False Positives\nWe understand that some teams may be flagged inadvertently. While we’ve removed thousands of teams from competitions, mistakes happen. If you believe your team was falsely identified and removed, we have have a simple appeals process:\n\n- **Submit a Ticket:** Visit [Kaggle Compliance](https://www.kaggle.com/compliance) and proceed with submitting a ticket if you believe you were falsely identified. We manually review every appeal. \n\nWe hope this provides some clarification on the actions we take to maintain fair play within our competitions.\n\n", "votes": 2, "replies": [ { "id": 3037451, "postDate": "2024-11-05T18:10:50.943Z", "content": "@jonathanmcwilliams thank you for your theoretical description of how all the processes should work.
\nI believe that Kaggle team has all the resources, knowledge and capacity available for following this itinerary.
\nBut …
\nDo you really claim that you managed all the described steps of post-competition screen (including manual check) in 20 hours respectively in 3 working hours (considering PDT) in both previous featured competitions?
Do you really push competitors to do your work instead of you (Reporting Cheating with concrete evidence) when all the evidence is fully in hands of the Kaggle team whose responsibility is to gather and evaluate it?
\nAnd should everything be really done secretly though it's in public interest?
Please follow my logic:
\nThe solutions (medals) sell-buy business exists. It is an indisputable fact and Kaggle team got enough precise evidence.
\nThe business wouldn't survive if it wasn't profitable.
\nIt wouldn't be profitable if potential medal \"winners\" weren't willing to pay significant amount of money for their \"product\".
\nThe buyers aren't foolish and wouldn't pay the money if they didn't have real chance for \"winning\" the medals based on their own or their friends' good experience with a previous successful purchase.
So some successful buyers must exist in the leaderboards. Where are they?
\nAren't they accidentally between places 43 and 65 in this competition and/or between places 40 and 72 in RSNA 2024 competition leaderboard?
\nIt's interesting that many members of those suspicious teams became Experts just after these 2 competitions.
Is Kaggle team doing enough to limit the demand for buying solutions thus financially cracking down on the business?
", "rawMarkdown": "@jonathanmcwilliams thank you for your theoretical description of how all the processes should work.\nI believe that Kaggle team has all the resources, knowledge and capacity available for following this itinerary.\nBut ...\nDo you really claim that you managed all the described steps of **post-competition screen** (including manual check) in 20 hours respectively in 3 working hours (considering PDT) in both previous featured competitions?\n \nDo you really push competitors to do your work instead of you (**Reporting Cheating with concrete evidence**) when all the evidence is fully in hands of the Kaggle team whose responsibility is to gather and evaluate it?\nAnd should everything be really done secretly though it's in public interest?\n\n**Please follow my logic:**\n\nThe solutions (medals) sell-buy business exists. It is an indisputable fact and Kaggle team got enough precise evidence.\n**The business wouldn't survive if it wasn't profitable.**\nIt wouldn't be profitable if potential medal \"winners\" weren't willing to pay significant amount of money for their \"product\".\nThe buyers aren't foolish and wouldn't pay the money if they didn't have real chance for \"winning\" the medals based on their own or their friends' good experience with a previous successful purchase.\n\nSo some **successful buyers must exist in the leaderboards**. Where are they?\nAren't they accidentally between places 43 and 65 in this competition and/or between places 40 and 72 in RSNA 2024 competition leaderboard? \n********************It's interesting that many members of those suspicious teams became Experts just after these 2 competitions.\n\nIs Kaggle team doing enough to limit the demand for buying solutions thus financially cracking down on the business?\n\n\n", "votes": 1 } ] } ], "comments": [ { "id": 3033759, "author_name": "greySnow", "author_url": "", "post_date": "2024-11-01T13:34:45.763000", "content": "Now it's purging time, lets wait until the LB finalize and see if the suspected teams are removed…
", "votes": 6, "replies": [] }, { "id": 3034171, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-11-01T21:11:15.037000", "content": "It really looks odd, especially since some of them participated in other competitions and also landed in a similar cluster… But it seems they haven't been removed. E.g. the RSNA competition, ranks 40-50:
\nhttps://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/leaderboard
\n@sohier was this actually investigated? I trust the kaggle team though, if they say this is all valid
As you mentioned the last RSNA competition - a similar discussion was led there too.
\nAnd both competitions were finalized too quickly (in less than 20 hours) to believe that any further investigation had been done.
I really don't understand the reason but the current Kaggle competitions administration seems to lack any will to crack down on this problem.
\nUnfortunately this situation devalues all medals and awards.
Maybe kaggle does not care about silver/bronze medals value since I see that the suspected teams are still there, and they are definitely suspicious.
", "votes": 0, "replies": [] }, { "id": 3034465, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2024-11-02T07:04:49.483000", "content": "That would be really sad. For many participants, winning bronze or silver medals is a great achievement and they invest many weeks of their free time in the competitions. This can't be right from the company's point of view either, because it could deter participants from taking part in Kaggle competitions in the future.
\nAny form of private sharing, cheating or purchased solutions should be excluded as far as possible (maybe that would be an interesting competition: predicting whether a solution was legit or not 😃 ). Personally, I'm fine with validation taking a few days (as in the past)…
", "votes": 7, "replies": [] } ] } ] }, { "id": 3036791, "author_name": "Jonathan McWilliams", "author_url": "", "post_date": "2024-11-04T23:22:20.143000", "content": "We would like to provide some insight into how we maintain fairness and integrity within our competitions. We screen competitions after the close (if done during the competition, teams could re-enter under new accounts), and because we do not publicly disclose our actions, there is sometimes a misconception that Kaggle ignores evidence of foul play. Kaggle does not overlook evidence of rules violations, nor give unequal treatment based on leaderboard rank, account tenure, progression status, or any other extenuating factor.
\nBelow is an overview of the steps we take to identify and address cheating and guidance on reporting any concerns.
\nBehind the scenes, our team monitors various activities, including leaderboards, submissions, user behavior, and team formation/merging. While we do not publicly address user-reported evidence of cheating, we are engaged with our community members who report suspicious activity. Details on how to report cheating are found below.
\nOnce a competition ends, our compliance team conducts an in-depth review, combining automated checks with manual oversight. Though we do not reveal the details of our review process, we listen to and have implemented community-sourced ideas to strengthen this process.
\nWe added an account verification process in early 2024 to deter cheating up front. Since adding this requirement, we have observed a marked drop in attempts to game competitions. Please see our announcement for more detail on the verification process.
\nWe encourage community members to report any suspicious activity or rule violations. Our compliance team evaluates each and every report.
\nTo report a concern, please:
\nSubmit a Ticket: Visit Support/Contact and navigate to \"I’d like to report cheating in a Competition.\" If you believe you have sufficient evidence that warrants an investigation, please proceed with submitting a ticket. We manually review every report received.
Provide Specific Details: Include team names, usernames, observed behaviors, screenshots, and, if available, screenshots or other supporting evidence.
All reports are handled confidentially. While we may not share the specific outcomes of each report, we value community input and use these to improve our effectiveness.
\nThe strength of evidence matters. It is not sufficient for Kaggle to take action on speculation or the mere appearance of wrongdoing. We must corroborate reports with concrete evidence of rules violations.
\nWe understand that some teams may be flagged inadvertently. While we’ve removed thousands of teams from competitions, mistakes happen. If you believe your team was falsely identified and removed, we have have a simple appeals process:
\nWe hope this provides some clarification on the actions we take to maintain fair play within our competitions.
", "votes": 2, "replies": [ { "id": 3037451, "author_name": "Allie K.", "author_url": "", "post_date": "2024-11-05T18:10:50.943000", "content": "@jonathanmcwilliams thank you for your theoretical description of how all the processes should work.
\nI believe that Kaggle team has all the resources, knowledge and capacity available for following this itinerary.
\nBut …
\nDo you really claim that you managed all the described steps of post-competition screen (including manual check) in 20 hours respectively in 3 working hours (considering PDT) in both previous featured competitions?
Do you really push competitors to do your work instead of you (Reporting Cheating with concrete evidence) when all the evidence is fully in hands of the Kaggle team whose responsibility is to gather and evaluate it?
\nAnd should everything be really done secretly though it's in public interest?
Please follow my logic:
\nThe solutions (medals) sell-buy business exists. It is an indisputable fact and Kaggle team got enough precise evidence.
\nThe business wouldn't survive if it wasn't profitable.
\nIt wouldn't be profitable if potential medal \"winners\" weren't willing to pay significant amount of money for their \"product\".
\nThe buyers aren't foolish and wouldn't pay the money if they didn't have real chance for \"winning\" the medals based on their own or their friends' good experience with a previous successful purchase.
So some successful buyers must exist in the leaderboards. Where are they?
\nAren't they accidentally between places 43 and 65 in this competition and/or between places 40 and 72 in RSNA 2024 competition leaderboard?
\nIt's interesting that many members of those suspicious teams became Experts just after these 2 competitions.
Is Kaggle team doing enough to limit the demand for buying solutions thus financially cracking down on the business?
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "3033743": "On the private leaderboard of this competition there are at least 18 teams with all the typical and many times described signs of involvement in the solutions (medals) sell-buy business. They are visibly clustered between places 43 and 65.\n \nRecently from all leaderboards of competitions without a significant shake up and from various public comments and testimonies it could be judged that the medals sell-buy business has been flourishing. \n\nSeveral times Kaggle team got a lot of well documented detailed information about how the business works from decent Kagglers (including Grandmasters) who speak the language. The links to the business websites were also published.\nAll the competitions are code competitions so it should be easy to compare the codes of submissions of visibly suspicious teams. Does Kaggle team routinely do it?\n\nI cannot say whether Kaggle team reacted anyhow internally to all the given evidence but the results were invisible and the leaderboards stayed nearly as they had been.\n\nAll this necessarily leads to the question:\n@wcukierski, @addisonhoward, @maggiemd in your opinion and according to Kaggle rules is buying a solution a newly authorized way how to participate in a Kaggle competition and get all possible awards?\n\n\n", "3033759": "Now it's purging time, lets wait until the LB finalize and see if the suspected teams are removed...", "3034171": "It really looks odd, especially since some of them participated in other competitions and also landed in a similar cluster... But it seems they haven't been removed. E.g. the RSNA competition, ranks 40-50:\nhttps://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/leaderboard\n@sohier was this actually investigated? I trust the kaggle team though, if they say this is all valid", "3036791": "We would like to provide some insight into how we maintain fairness and integrity within our competitions. We screen competitions after the close (if done during the competition, teams could re-enter under new accounts), and because we do not publicly disclose our actions, there is sometimes a misconception that Kaggle ignores evidence of foul play. Kaggle does not overlook evidence of rules violations, nor give unequal treatment based on leaderboard rank, account tenure, progression status, or any other extenuating factor.\n\nBelow is an overview of the steps we take to identify and address cheating and guidance on reporting any concerns.\n\n# Active Monitoring\nBehind the scenes, our team monitors various activities, including leaderboards, submissions, user behavior, and team formation/merging. While we do not publicly address user-reported evidence of cheating, we are engaged with our community members who report suspicious activity. Details on how to report cheating are found below.\n\n# Post-Competition Screen \nOnce a competition ends, our compliance team conducts an in-depth review, combining automated checks with manual oversight. Though we do not reveal the details of our review process, we listen to and have implemented community-sourced ideas to strengthen this process.\n\n# Verification\nWe added an account verification process in early 2024 to deter cheating up front. Since adding this requirement, we have observed a marked drop in attempts to game competitions. Please see our [announcement](https://www.kaggle.com/discussions/general/490841) for more detail on the verification process.\n\n# Reporting Cheating\n**We encourage community members to report any suspicious activity or rule violations.** Our compliance team evaluates each and every report.\n\nTo report a concern, please:\n\n- **Submit a Ticket:** Visit [Support/Contact](https://www.kaggle.com/contact) and navigate to \"I’d like to report cheating in a Competition.\" If you believe you have sufficient evidence that warrants an investigation, please proceed with submitting a ticket. We manually review every report received. \n\n- **Provide Specific Details:** Include team names, usernames, observed behaviors, screenshots, and, if available, screenshots or other supporting evidence.\n\nAll reports are handled confidentially. While we may not share the specific outcomes of each report, we value community input and use these to improve our effectiveness.\n\n**The strength of evidence matters.** It is not sufficient for Kaggle to take action on speculation or the mere appearance of wrongdoing. We must corroborate reports with concrete evidence of rules violations.\n\n# Appeals and Safeguards Against False Positives\nWe understand that some teams may be flagged inadvertently. While we’ve removed thousands of teams from competitions, mistakes happen. If you believe your team was falsely identified and removed, we have have a simple appeals process:\n\n- **Submit a Ticket:** Visit [Kaggle Compliance](https://www.kaggle.com/compliance) and proceed with submitting a ticket if you believe you were falsely identified. We manually review every appeal. \n\nWe hope this provides some clarification on the actions we take to maintain fair play within our competitions.\n\n" } }