To prevent a fiasco where one of my selected submissions violated the label consistency requirements, I checked to make sure that my submission file had 0 inconsistent predictions using the host-provided check_consistency function.
I was checking label consistency of my submissions after generating the submission file (by loading it back in and running the function), with an assertion statement that the # of errors is zero.
\nThis morning, I realized that this could still result in a successful submission. Even if the assertion throws an error, submission.csv is already generated- so I believe the submission would still be successful.
So, now I'm just hoping my function to correct all the inconsistent predictions has no bugs since there's not enough time in the competition for me to fix it. My most recent commit took 3x longer than usual, likely because of all the submission traffic going on right nowβ¦ π
", "messages": [ { "id": 1060743, "postDate": "2020-10-26T14:20:54.287Z", "content": "To prevent a fiasco where one of my selected submissions violated the label consistency requirements, I checked to make sure that my submission file had 0 inconsistent predictions using the host-provided check_consistency function.
I was checking label consistency of my submissions after generating the submission file (by loading it back in and running the function), with an assertion statement that the # of errors is zero.
\nThis morning, I realized that this could still result in a successful submission. Even if the assertion throws an error, submission.csv is already generated- so I believe the submission would still be successful.
So, now I'm just hoping my function to correct all the inconsistent predictions has no bugs since there's not enough time in the competition for me to fix it. My most recent commit took 3x longer than usual, likely because of all the submission traffic going on right nowβ¦ π
", "rawMarkdown": "To prevent a fiasco where one of my selected submissions violated the label consistency requirements, I checked to make sure that my submission file had 0 inconsistent predictions using the host-provided `check_consistency` function. \n\nI was checking label consistency of my submissions **after** generating the submission file (by loading it back in and running the function), with an assertion statement that the # of errors is zero.\n\nThis morning, I realized that this could still result in a successful submission. Even if the assertion throws an error, `submission.csv` is already generated- so I believe the submission would still be successful. \n\nSo, now I'm just hoping my function to correct all the inconsistent predictions has no bugs since there's not enough time in the competition for me to fix it. My most recent commit took 3x longer than usual, likely because of all the submission traffic going on right now... π ", "votes": 7 }, { "id": 1061227, "postDate": "2020-10-26T20:44:45.047Z", "content": "wishing you good luck in this @vaillant Your insights were great, and you got hit by enough bad luck in the prostate competition, so you should get good luck now πͺπ
\nI am finding the checking code rules preventing most of my submissions getting loaded - Anything which did not get a prediction I use the default submission file which is 0.5 for all records. In hindsight it would have been better to set the default values to 0.4999, as I keep hitting errors where I checked for <0.5 but checking code checks for <= 0.5. But I still think close to my best effort is in.
Edit⦠Actually, just two of my last 9 submissions made it thru cause of this stupid thing
", "rawMarkdown": "wishing you good luck in this @vaillant Your insights were great, and you got hit by enough bad luck in the prostate competition, so you should get good luck now πͺπ\nI am finding the checking code rules preventing most of my submissions getting loaded - Anything which did not get a prediction I use the default submission file which is 0.5 for all records. In hindsight it would have been better to set the default values to 0.4999, as I keep hitting errors where I checked for <0.5 but checking code checks for <= 0.5. But I still think close to my best effort is in. \n\nEdit... Actually, just two of my last 9 submissions made it thru cause of this stupid thing", "votes": 1 }, { "id": 1060755, "postDate": "2020-10-26T14:29:15.877Z", "content": "@vaillant I can tell we had the same problem regarding consistency correction, more than 24hrs ago. In my opinion it crashed due to Kaggle Traffict or Memory Problems (at the end, before consistency check, RAM was > 11Gb) we optimized everything and now we can check consistency in 400ms (all the studies).
", "rawMarkdown": "@vaillant I can tell we had the same problem regarding consistency correction, more than 24hrs ago. In my opinion it crashed due to Kaggle Traffict or Memory Problems (at the end, before consistency check, RAM was > 11Gb) we optimized everything and now we can check consistency in 400ms (all the studies)." } ], "comments": [ { "id": 1061227, "author_name": "Darragh", "author_url": "", "post_date": "2020-10-26T20:44:45.047000", "content": "wishing you good luck in this @vaillant Your insights were great, and you got hit by enough bad luck in the prostate competition, so you should get good luck now πͺπ
\nI am finding the checking code rules preventing most of my submissions getting loaded - Anything which did not get a prediction I use the default submission file which is 0.5 for all records. In hindsight it would have been better to set the default values to 0.4999, as I keep hitting errors where I checked for <0.5 but checking code checks for <= 0.5. But I still think close to my best effort is in.
Edit⦠Actually, just two of my last 9 submissions made it thru cause of this stupid thing
", "votes": 1, "replies": [] }, { "id": 1060755, "author_name": "Nanashi", "author_url": "", "post_date": "2020-10-26T14:29:15.877000", "content": "@vaillant I can tell we had the same problem regarding consistency correction, more than 24hrs ago. In my opinion it crashed due to Kaggle Traffict or Memory Problems (at the end, before consistency check, RAM was > 11Gb) we optimized everything and now we can check consistency in 400ms (all the studies).
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1060743": "To prevent a fiasco where one of my selected submissions violated the label consistency requirements, I checked to make sure that my submission file had 0 inconsistent predictions using the host-provided `check_consistency` function. \n\nI was checking label consistency of my submissions **after** generating the submission file (by loading it back in and running the function), with an assertion statement that the # of errors is zero.\n\nThis morning, I realized that this could still result in a successful submission. Even if the assertion throws an error, `submission.csv` is already generated- so I believe the submission would still be successful. \n\nSo, now I'm just hoping my function to correct all the inconsistent predictions has no bugs since there's not enough time in the competition for me to fix it. My most recent commit took 3x longer than usual, likely because of all the submission traffic going on right now... π ", "1061227": "wishing you good luck in this @vaillant Your insights were great, and you got hit by enough bad luck in the prostate competition, so you should get good luck now πͺπ\nI am finding the checking code rules preventing most of my submissions getting loaded - Anything which did not get a prediction I use the default submission file which is 0.5 for all records. In hindsight it would have been better to set the default values to 0.4999, as I keep hitting errors where I checked for <0.5 but checking code checks for <= 0.5. But I still think close to my best effort is in. \n\nEdit... Actually, just two of my last 9 submissions made it thru cause of this stupid thing", "1060755": "@vaillant I can tell we had the same problem regarding consistency correction, more than 24hrs ago. In my opinion it crashed due to Kaggle Traffict or Memory Problems (at the end, before consistency check, RAM was > 11Gb) we optimized everything and now we can check consistency in 400ms (all the studies)." } }