![\"pred-true\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F9b69f4649db2e2c21cc5425d08a63e02%2Fytp.png?generation=1722708074983145&alt=media\")
My notebook has a decent cv score but scores 0.000 on the leaderboard.
\n
Possible causes:
\nWhoever finds the bug will immediately be rewarded with a high leaderboard rank…
\nEDIT: It looks like the bug disappears as soon as training and inference code are separated into two different notebooks. The two notebooks ADC24 Intro training ⭐️⭐️⭐️⭐️⭐️ and ADC24 Intro inference ⭐️⭐️⭐️⭐️⭐️ show how it's done.
", "messages": [ { "id": 2945760, "postDate": "2024-08-03T18:02:15.047Z", "content": "My notebook has a decent cv score but scores 0.000 on the leaderboard.
\n
Possible causes:
\nWhoever finds the bug will immediately be rewarded with a high leaderboard rank…
\nEDIT: It looks like the bug disappears as soon as training and inference code are separated into two different notebooks. The two notebooks ADC24 Intro training ⭐️⭐️⭐️⭐️⭐️ and ADC24 Intro inference ⭐️⭐️⭐️⭐️⭐️ show how it's done.
", "rawMarkdown": "[My notebook](https://www.kaggle.com/code/ambrosm/adc24-quickstart) has a decent cv score but scores 0.000 on the leaderboard. \n\n\n\nPossible causes:\n1. My code contains a bug.\n2. The leaderboard scoring code contains a bug.\n\nWhoever finds the bug will immediately be rewarded with a high leaderboard rank...\n\nEDIT: It looks like the bug disappears as soon as training and inference code are separated into two different notebooks. The two notebooks *ADC24 Intro training ⭐️⭐️⭐️⭐️⭐️* and *ADC24 Intro inference ⭐️⭐️⭐️⭐️⭐️* show how it's done.", "votes": 24 }, { "id": 2945935, "postDate": "2024-08-03T20:51:56.993Z", "content": "\n\nSince Test set ~ 200GB size, maybe do not have Train dataset files or corrupted files (my guess about your notebook for getting 0 score even cv correlate with targets well - public notebook).
\n
\n", "rawMarkdown": "# [ADC24 Quickstart ⭐️⭐️⭐️⭐️⭐️]( https://www.kaggle.com/code/ambrosm/adc24-quickstart ) => **Public LB: 0.182** @ambrosm \n\n---\n\n> Since Test set ~ 200GB size, maybe do not have Train dataset files or corrupted files (my guess about your notebook for getting 0 score even cv correlate with targets well - public notebook). \n\n---\n\n> **Submitted inference notebook with your trained model and supporting files.** => **Public LB: 0.182 / CV: 0.389**\n-- LB & CV gap can be domain shift or scoring bug or issue in my submission notebook !!\n\n", "votes": 5, "replies": [ { "id": 2948673, "postDate": "2024-08-06T03:14:51.160Z", "content": "Submitted inference notebook with your trained model and supporting files. => Public LB: 0.182 / CV: 0.389
\n
\n -- LB & CV gap can be domain shift or scoring bug or issue in my submission notebook !!
Hi @seshurajup, thank you for the hint about training files being unavailable during the submission! It looks like everything works ok after separating the code into training and inference notebooks so that the inference notebook doesn't depend on the training data.
", "rawMarkdown": "Hi @seshurajup, thank you for the hint about training files being unavailable during the submission! It looks like everything works ok after separating the code into training and inference notebooks so that the inference notebook doesn't depend on the training data.", "votes": 3, "replies": [ { "id": 2948678, "postDate": "2024-08-06T03:18:14.930Z", "content": "@ambrosm - i'm playing with metric, choice of sigma value also leading to zero score. Do you know how to handle the sigma targets? as competition is more sensitive to the sigma targets. Thanks for sharing your magic feature.
\n\n", "rawMarkdown": "@ambrosm - i'm playing with metric, choice of sigma value also leading to zero score. Do you know how to handle the sigma targets? as competition is more sensitive to the sigma targets. Thanks for sharing your **magic feature**.\n\n> Gap between CV and LB is not matching well " }, { "id": 2957174, "postDate": "2024-08-12T21:10:37.317Z", "content": "Gap between CV and LB is not matching well
\n
I noticed that separating training and inference into different notebooks resolved the scoring issue in this case However, doesn't this approach potentially violate Kaggle's rules for private notebooks? My understanding is that any model used during inference must either be trained in the same notebook or made publicly accessible if trained separately. Could someone clarify whether this separation is allowed in private notebooks without making the trained model public?
", "rawMarkdown": "I noticed that separating training and inference into different notebooks resolved the scoring issue in this case However, doesn't this approach potentially violate Kaggle's rules for private notebooks? My understanding is that any model used during inference must either be trained in the same notebook or made publicly accessible if trained separately. Could someone clarify whether this separation is allowed in private notebooks without making the trained model public?\n\n" }, { "id": 2957265, "postDate": "2024-08-13T00:56:05.677Z", "content": "Your understanding is wrong sravn.
\nModel can be trained anywhere with anything. Only the prediction needs to be made in a notebook.
\nThe 'publicly accessible' has always referred to any additional data that you find to train the model.
", "rawMarkdown": "Your understanding is wrong [sravn](https://www.kaggle.com/sorenravn).\n\nModel can be trained anywhere with anything. Only the prediction needs to be made in a notebook.\n\nThe 'publicly accessible' has always referred to any additional data that you find to train the model.", "votes": 2 } ] } ] }, { "id": 2951397, "postDate": "2024-08-08T15:54:27.910Z", "content": "Facing the same issue. I think the main problem is that we don't know the sigma_true value. Adding
\nL = float(np.clip(submit_score, 0.0, 1.0))
\nprint(f\"{sigma_true=}\\n{GLL_pred=}\\n{GLL_true=}\\n{GLL_mean=}\\n{L=}\\n----\\n\")
\nto the score function, and trying different sigma values, I have noticed that for sigma_true>0.003 GLL_true(L_ideal) is less than GLL_mean(L_ref), which leads to a negative value in the denominator, and since the score is clipped to not accept negative values it returns 0.
Hi @arkano29 , just trying to better understand the issue - the sigma_true value is set as 10ppm (1e-5) in the metric section, so i dont think it will turn into a negative value during score calculation.
", "rawMarkdown": "Hi @arkano29 , just trying to better understand the issue - the sigma_true value is set as 10ppm (1e-5) in the metric section, so i dont think it will turn into a negative value during score calculation. ", "votes": 6, "replies": [ { "id": 2953981, "postDate": "2024-08-09T08:59:35.947Z", "content": "Ok, then it's probably that my model is overfitted or something else, since it gets a good score in train but 0 in test. Thank you!
", "rawMarkdown": "Ok, then it's probably that my model is overfitted or something else, since it gets a good score in train but 0 in test. Thank you!" }, { "id": 2954320, "postDate": "2024-08-09T15:22:08.343Z", "content": "it could happen, certain aspects of the test data distribution are deliberately made different from the training set. Please see welcome post for more info.
", "rawMarkdown": "it could happen, certain aspects of the test data distribution are deliberately made different from the training set. Please see welcome post for more info. ", "votes": 1 } ] }, { "id": 2954003, "postDate": "2024-08-09T09:29:09.293Z", "content": "hi @arkano29 For me the issue disappeared when I separated training and inference code into two notebooks (ADC24 Intro training and ADC24 Intro inference). Perhaps during submission the notebook no longer has access to the full training data.
", "rawMarkdown": "hi @arkano29 For me the issue disappeared when I separated training and inference code into two notebooks (ADC24 Intro training and ADC24 Intro inference). Perhaps during submission the notebook no longer has access to the full training data.\n\n", "votes": 3, "replies": [ { "id": 2954083, "postDate": "2024-08-09T11:13:53.073Z", "content": "Thank you! I will try that.
", "rawMarkdown": "Thank you! I will try that." } ] } ] }, { "id": 2958632, "postDate": "2024-08-14T06:55:04.827Z", "content": "hii , @ambrosm
\nDid you tried to train the model with both the uncalibrated data and calibrated data?
\nDid you find any improvements in one over other?
Hi @smitten20,
\nSome questions are still open about calibration, and I wanted to first see the big picture. Calibration is on my to do list.
", "rawMarkdown": "Hi @smitten20,\n\n[Some questions](https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/527644) are still open about calibration, and I wanted to first see the big picture. Calibration is on my to do list." } ] }, { "id": 2954555, "postDate": "2024-08-09T20:38:19.067Z", "content": "I will win if I found the bug?
", "rawMarkdown": "I will win if I found the bug?" } ], "comments": [ { "id": 2945935, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2024-08-03T20:51:56.993000", "content": "\n\nSince Test set ~ 200GB size, maybe do not have Train dataset files or corrupted files (my guess about your notebook for getting 0 score even cv correlate with targets well - public notebook).
\n
\n", "votes": 5, "replies": [ { "id": 2948673, "author_name": "AmbrosM", "author_url": "", "post_date": "2024-08-06T03:14:51.160000", "content": "Submitted inference notebook with your trained model and supporting files. => Public LB: 0.182 / CV: 0.389
\n
\n -- LB & CV gap can be domain shift or scoring bug or issue in my submission notebook !!
Hi @seshurajup, thank you for the hint about training files being unavailable during the submission! It looks like everything works ok after separating the code into training and inference notebooks so that the inference notebook doesn't depend on the training data.
", "votes": 3, "replies": [ { "id": 2948678, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2024-08-06T03:18:14.930000", "content": "@ambrosm - i'm playing with metric, choice of sigma value also leading to zero score. Do you know how to handle the sigma targets? as competition is more sensitive to the sigma targets. Thanks for sharing your magic feature.
\n\n", "votes": 0, "replies": [] }, { "id": 2957174, "author_name": "Søren Ravn Andersen", "author_url": "", "post_date": "2024-08-12T21:10:37.317000", "content": "Gap between CV and LB is not matching well
\n
I noticed that separating training and inference into different notebooks resolved the scoring issue in this case However, doesn't this approach potentially violate Kaggle's rules for private notebooks? My understanding is that any model used during inference must either be trained in the same notebook or made publicly accessible if trained separately. Could someone clarify whether this separation is allowed in private notebooks without making the trained model public?
", "votes": 0, "replies": [] }, { "id": 2957265, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2024-08-13T00:56:05.677000", "content": "Your understanding is wrong sravn.
\nModel can be trained anywhere with anything. Only the prediction needs to be made in a notebook.
\nThe 'publicly accessible' has always referred to any additional data that you find to train the model.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2951397, "author_name": "Arkaitz", "author_url": "", "post_date": "2024-08-08T15:54:27.910000", "content": "Facing the same issue. I think the main problem is that we don't know the sigma_true value. Adding
\nL = float(np.clip(submit_score, 0.0, 1.0))
\nprint(f\"{sigma_true=}\\n{GLL_pred=}\\n{GLL_true=}\\n{GLL_mean=}\\n{L=}\\n----\\n\")
\nto the score function, and trying different sigma values, I have noticed that for sigma_true>0.003 GLL_true(L_ideal) is less than GLL_mean(L_ref), which leads to a negative value in the denominator, and since the score is clipped to not accept negative values it returns 0.
Hi @arkano29 , just trying to better understand the issue - the sigma_true value is set as 10ppm (1e-5) in the metric section, so i dont think it will turn into a negative value during score calculation.
", "votes": 6, "replies": [ { "id": 2953981, "author_name": "Arkaitz", "author_url": "", "post_date": "2024-08-09T08:59:35.947000", "content": "Ok, then it's probably that my model is overfitted or something else, since it gets a good score in train but 0 in test. Thank you!
", "votes": 0, "replies": [] }, { "id": 2954320, "author_name": "Gordon Yip", "author_url": "", "post_date": "2024-08-09T15:22:08.343000", "content": "it could happen, certain aspects of the test data distribution are deliberately made different from the training set. Please see welcome post for more info.
", "votes": 1, "replies": [] } ] }, { "id": 2954003, "author_name": "AmbrosM", "author_url": "", "post_date": "2024-08-09T09:29:09.293000", "content": "hi @arkano29 For me the issue disappeared when I separated training and inference code into two notebooks (ADC24 Intro training and ADC24 Intro inference). Perhaps during submission the notebook no longer has access to the full training data.
", "votes": 3, "replies": [ { "id": 2954083, "author_name": "Arkaitz", "author_url": "", "post_date": "2024-08-09T11:13:53.073000", "content": "Thank you! I will try that.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2958632, "author_name": "smittenk", "author_url": "", "post_date": "2024-08-14T06:55:04.827000", "content": "hii , @ambrosm
\nDid you tried to train the model with both the uncalibrated data and calibrated data?
\nDid you find any improvements in one over other?
Hi @smitten20,
\nSome questions are still open about calibration, and I wanted to first see the big picture. Calibration is on my to do list.
", "votes": 0, "replies": [] } ] }, { "id": 2954555, "author_name": "Simo Medamro", "author_url": "", "post_date": "2024-08-09T20:38:19.067000", "content": "I will win if I found the bug?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2945760": "[My notebook](https://www.kaggle.com/code/ambrosm/adc24-quickstart) has a decent cv score but scores 0.000 on the leaderboard. \n\n\n\nPossible causes:\n1. My code contains a bug.\n2. The leaderboard scoring code contains a bug.\n\nWhoever finds the bug will immediately be rewarded with a high leaderboard rank...\n\nEDIT: It looks like the bug disappears as soon as training and inference code are separated into two different notebooks. The two notebooks *ADC24 Intro training ⭐️⭐️⭐️⭐️⭐️* and *ADC24 Intro inference ⭐️⭐️⭐️⭐️⭐️* show how it's done.", "2945935": "# [ADC24 Quickstart ⭐️⭐️⭐️⭐️⭐️]( https://www.kaggle.com/code/ambrosm/adc24-quickstart ) => **Public LB: 0.182** @ambrosm \n\n---\n\n> Since Test set ~ 200GB size, maybe do not have Train dataset files or corrupted files (my guess about your notebook for getting 0 score even cv correlate with targets well - public notebook). \n\n---\n\n> **Submitted inference notebook with your trained model and supporting files.** => **Public LB: 0.182 / CV: 0.389**\n-- LB & CV gap can be domain shift or scoring bug or issue in my submission notebook !!\n\n", "2951397": "Facing the same issue. I think the main problem is that we don't know the sigma_true value. Adding \n`L = float(np.clip(submit_score, 0.0, 1.0))` \n`print(f\"{sigma_true=}\\n{GLL_pred=}\\n{GLL_true=}\\n{GLL_mean=}\\n{L=}\\n----\\n\")` \nto the score function, and trying different sigma values, I have noticed that **for sigma_true>0.003 GLL_true(L_ideal) is less than GLL_mean(L_ref)**, which leads to a negative value in the denominator, and since the score is clipped to not accept negative values it returns 0.", "2958632": "hii , @ambrosm \nDid you tried to train the model with both the uncalibrated data and calibrated data?\nDid you find any improvements in one over other? ", "2954555": "I will win if I found the bug?" } }