Would it be against the rules to re-label any of the submission data if I think my network output is slightly wrong? For example, if my network output gives 0.9 to sample number x, I would write a function that tries to target sample x and any others like it and adjust its score.
\nI am asking this because a few samples in the test set have very awkward noise, unique to any data in the training set. I'm guessing that this noise is real, and it would be difficult to make similar data myself to add to a training set. It's especially annoying because in some of these awkward noise test set samples, it is obvious to me (by eye) that these samples have CWs and should get 1.0.
Here's the general rule: Whatever you do has to be generalizable to unseen data.
\nYou can, e.g., calculate a noise factor and adjust model predictions based on that. You cannot add something to your code that makes an adjustment for a particular observation based on its id (or some proxy that is effectively identifying a specific observation). We should be able to completely rename all of the test data, and your model should give the exact same result. And the model should give similar results on completely new test data.
Makes sense?
\n(BTW, I'm basically \"up the street\" from you in Neenah, WI).
", "rawMarkdown": "Here's the general rule: Whatever you do has to be generalizable to unseen data. \n\nYou can, e.g., calculate a noise factor and adjust model predictions based on that. You _cannot_ add something to your code that makes an adjustment for a particular observation based on its `id` (or some proxy that is effectively identifying a specific observation). We should be able to completely rename all of the test data, and your model should give the exact same result. And the model should give similar results on completely new test data.\n\nMakes sense?\n\n(BTW, I'm basically \"up the street\" from you in Neenah, WI).", "votes": 1 }, { "id": 2016740, "postDate": "2022-11-04T07:58:04.210Z", "content": "I'm not aware that post processing in a general way is illegal, you could just set a simple threshold and label all samples with probabilities above >0.8 as 1 this is absolutely your decision how you handle the output of your model.
\nIn case of labeling unique samples the rule I found is:
\nSubmissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.
Yes, that illegal.
\nAlso I think that is part of the reason why the test set is so big, yet final results based only on a subset of it ( so that human labeling attempts become difficult).
", "rawMarkdown": "Yes, that illegal.\n\nAlso I think that is part of the reason why the test set is so big, yet final results based only on a subset of it ( so that human labeling attempts become difficult)." }, { "id": 2016666, "postDate": "2022-11-04T07:08:14.300Z", "content": "Would it be against the rules to re-label any of the submission data if I think my network output is slightly wrong? For example, if my network output gives 0.9 to sample number x, I would write a function that tries to target sample x and any others like it and adjust its score.
\nI am asking this because a few samples in the test set have very awkward noise, unique to any data in the training set. I'm guessing that this noise is real, and it would be difficult to make similar data myself to add to a training set. It's especially annoying because in some of these awkward noise test set samples, it is obvious to me (by eye) that these samples have CWs and should get 1.0.
Here's the general rule: Whatever you do has to be generalizable to unseen data.
\nYou can, e.g., calculate a noise factor and adjust model predictions based on that. You cannot add something to your code that makes an adjustment for a particular observation based on its id (or some proxy that is effectively identifying a specific observation). We should be able to completely rename all of the test data, and your model should give the exact same result. And the model should give similar results on completely new test data.
Makes sense?
\n(BTW, I'm basically \"up the street\" from you in Neenah, WI).
", "votes": 1, "replies": [] }, { "id": 2016740, "author_name": "Ali Abdin", "author_url": "", "post_date": "2022-11-04T07:58:04.210000", "content": "I'm not aware that post processing in a general way is illegal, you could just set a simple threshold and label all samples with probabilities above >0.8 as 1 this is absolutely your decision how you handle the output of your model.
\nIn case of labeling unique samples the rule I found is:
\nSubmissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.
Yes, that illegal.
\nAlso I think that is part of the reason why the test set is so big, yet final results based only on a subset of it ( so that human labeling attempts become difficult).
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2017507": "Here's the general rule: Whatever you do has to be generalizable to unseen data. \n\nYou can, e.g., calculate a noise factor and adjust model predictions based on that. You _cannot_ add something to your code that makes an adjustment for a particular observation based on its `id` (or some proxy that is effectively identifying a specific observation). We should be able to completely rename all of the test data, and your model should give the exact same result. And the model should give similar results on completely new test data.\n\nMakes sense?\n\n(BTW, I'm basically \"up the street\" from you in Neenah, WI).", "2016740": "I'm not aware that post processing in a general way is illegal, you could just set a simple threshold and label all samples with probabilities above >0.8 as 1 this is absolutely your decision how you handle the output of your model.\n\nIn case of labeling unique samples the rule I found is:\n\n`Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.`", "2017451": "Yes, that illegal.\n\nAlso I think that is part of the reason why the test set is so big, yet final results based only on a subset of it ( so that human labeling attempts become difficult).", "2016666": "Would it be against the rules to re-label any of the submission data if I think my network output is slightly wrong? For example, if my network output gives 0.9 to sample number x, I would write a function that tries to target sample x and any others like it and adjust its score.\nI am asking this because a few samples in the test set have very awkward noise, unique to any data in the training set. I'm guessing that this noise is real, and it would be difficult to make similar data myself to add to a training set. It's especially annoying because in some of these awkward noise test set samples, it is obvious to me (by eye) that these samples have CWs and should get 1.0." } }