It seems that my nn model gives a reasonable cv score (>0.55) across kfold, but the LB score is always 0. What could be the reason? Could it be that my model give a prediction too big because the distribution of test set is different?
", "messages": [ { "id": 3012480, "postDate": "2024-10-09T05:02:45.173Z", "content": "I struggled with 0 score for weeks. In my case the key difference between train and test was probably that the ingress and egress times (i.e. when the planet enters and exits) are different in test. So if you're making specific assumptions on when this happens (for example by removing a large amount of data around the possible ingress and egress), you can run into this.
", "rawMarkdown": "I struggled with 0 score for weeks. In my case the key difference between train and test was probably that the ingress and egress times (i.e. when the planet enters and exits) are different in test. So if you're making specific assumptions on when this happens (for example by removing a large amount of data around the possible ingress and egress), you can run into this.", "votes": 12, "replies": [ { "id": 3012507, "postDate": "2024-10-09T05:49:20.093Z", "content": "π thanks a lot, I did make such assumption. The training set has this period well centered.
", "rawMarkdown": "π thanks a lot, I did make such assumption. The training set has this period well centered." }, { "id": 3012760, "postDate": "2024-10-09T10:44:22.583Z", "content": "Do you know if the test set's dataset well centered? (ingress between [0, 188], egress between [188, 375])
\nIt is really frustating debugging with small amount of submissions and no score feedback β¦ the cv score is quite good but lb always 0 :/
I can confirm that there are cases when some examples in the test is not as well-centered as others. If you look into the training set, there are also a few cases where it is not well-centered too.
", "rawMarkdown": "I can confirm that there are cases when some examples in the test is not as well-centered as others. If you look into the training set, there are also a few cases where it is not well-centered too. ", "votes": 3 }, { "id": 3015802, "postDate": "2024-10-13T00:01:01.873Z", "content": "How did you manage to fix the 0 score LB? Was it related to transit timing?
", "rawMarkdown": "How did you manage to fix the 0 score LB? Was it related to transit timing?" }, { "id": 3031438, "postDate": "2024-10-29T17:25:52.810Z", "content": "did you find out?
", "rawMarkdown": "did you find out?\n" } ] } ] }, { "id": 3012413, "postDate": "2024-10-09T02:56:54.960Z", "content": "It seems that my nn model gives a reasonable cv score (>0.55) across kfold, but the LB score is always 0. What could be the reason? Could it be that my model give a prediction too big because the distribution of test set is different?
", "rawMarkdown": "It seems that my nn model gives a reasonable cv score (>0.55) across kfold, but the LB score is always 0. What could be the reason? Could it be that my model give a prediction too big because the distribution of test set is different? ", "votes": 9 }, { "id": 3012438, "postDate": "2024-10-09T03:27:03.467Z", "content": "Try increasing sigma prediction. Uncertainty for test set predictions is much higher and caused people to have zero score
", "rawMarkdown": "Try increasing sigma prediction. Uncertainty for test set predictions is much higher and caused people to have zero score", "votes": 3, "replies": [ { "id": 3012456, "postDate": "2024-10-09T04:03:59.627Z", "content": "Thanks, it seems that my model does not generalize when the distribution changes.
", "rawMarkdown": "Thanks, it seems that my model does not generalize when the distribution changes.", "votes": 1 } ] }, { "id": 3022135, "postDate": "2024-10-19T10:11:36.870Z", "content": "Also encountered this problem, would you share your idea on how to resolve it?
", "rawMarkdown": "Also encountered this problem, would you share your idea on how to resolve it?", "replies": [ { "id": 3024047, "postDate": "2024-10-21T09:27:02.420Z", "content": "Increase sigma value to like 5e-4
", "rawMarkdown": "Increase sigma value to like 5e-4", "votes": 1 } ] }, { "id": 3014283, "postDate": "2024-10-11T05:42:03.297Z", "content": "I used a very simple model, local cv is ~0.7, online score is much lower. Is there anything missing that I did not noticeβ¦ I do not think I would be able to figure it out before the end of the competition β¦π
", "rawMarkdown": "I used a very simple model, local cv is ~0.7, online score is much lower. Is there anything missing that I did not notice... I do not think I would be able to figure it out before the end of the competition ...π ", "replies": [ { "id": 3014303, "postDate": "2024-10-11T06:20:11.640Z", "content": "It's the same here. I'm considering the following:
\nTo be honest, they should make the only difference between train/test data the new star. Currently I do not have any idea how to optimize β¦ only submit & zeros
", "rawMarkdown": "To be honest, they should make the only difference between train/test data the new star. Currently I do not have any idea how to optimize ... only submit & zeros" }, { "id": 3014493, "postDate": "2024-10-11T09:27:08.810Z", "content": "Just so you're aware: there are two new stars (so 4 stars total in the test set). You can verify this with an appropriate assert in a submission script.
", "rawMarkdown": "Just so you're aware: there are two new stars (so 4 stars total in the test set). You can verify this with an appropriate assert in a submission script.", "votes": 4 }, { "id": 3014494, "postDate": "2024-10-11T09:27:19.670Z", "content": "@wuliaokaola Hello, Johnny. what do you mean 'The new star (=2)'? Can we probe the test dataset?
", "rawMarkdown": "@wuliaokaola Hello, Johnny. what do you mean 'The new star (=2)'? Can we probe the test dataset?" }, { "id": 3014558, "postDate": "2024-10-11T10:54:03.837Z", "content": "50% data for star0/1, 50% data for star2/3
", "rawMarkdown": "50% data for star0/1, 50% data for star2/3", "votes": 1 }, { "id": 3023537, "postDate": "2024-10-20T16:56:48.133Z", "content": "Hi, how did you finally resolve the zero score error?
", "rawMarkdown": "Hi, how did you finally resolve the zero score error?" } ] }, { "id": 3014462, "postDate": "2024-10-11T09:10:06.263Z", "content": "That's normal and I also meet the same situation. It's easy to train and predict the wavelengths, and I try finding the different pre- or proprocessing for test datasets.
", "rawMarkdown": "That's normal and I also meet the same situation. It's easy to train and predict the wavelengths, and I try finding the different pre- or proprocessing for test datasets." }, { "id": 3015477, "postDate": "2024-10-12T13:22:52.197Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 3012480, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2024-10-09T05:02:45.173000", "content": "I struggled with 0 score for weeks. In my case the key difference between train and test was probably that the ingress and egress times (i.e. when the planet enters and exits) are different in test. So if you're making specific assumptions on when this happens (for example by removing a large amount of data around the possible ingress and egress), you can run into this.
", "votes": 12, "replies": [ { "id": 3012507, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-10-09T05:49:20.093000", "content": "π thanks a lot, I did make such assumption. The training set has this period well centered.
", "votes": 0, "replies": [] }, { "id": 3012760, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-10-09T10:44:22.583000", "content": "Do you know if the test set's dataset well centered? (ingress between [0, 188], egress between [188, 375])
\nIt is really frustating debugging with small amount of submissions and no score feedback β¦ the cv score is quite good but lb always 0 :/
I can confirm that there are cases when some examples in the test is not as well-centered as others. If you look into the training set, there are also a few cases where it is not well-centered too.
", "votes": 3, "replies": [] }, { "id": 3015802, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2024-10-13T00:01:01.873000", "content": "How did you manage to fix the 0 score LB? Was it related to transit timing?
", "votes": 0, "replies": [] }, { "id": 3031438, "author_name": "highDopamine", "author_url": "", "post_date": "2024-10-29T17:25:52.810000", "content": "did you find out?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3012438, "author_name": "snehal", "author_url": "", "post_date": "2024-10-09T03:27:03.467000", "content": "Try increasing sigma prediction. Uncertainty for test set predictions is much higher and caused people to have zero score
", "votes": 3, "replies": [ { "id": 3012456, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-10-09T04:03:59.627000", "content": "Thanks, it seems that my model does not generalize when the distribution changes.
", "votes": 1, "replies": [] } ] }, { "id": 3022135, "author_name": "ZHANG XINGHAN", "author_url": "", "post_date": "2024-10-19T10:11:36.870000", "content": "Also encountered this problem, would you share your idea on how to resolve it?
", "votes": 0, "replies": [ { "id": 3024047, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-10-21T09:27:02.420000", "content": "Increase sigma value to like 5e-4
", "votes": 1, "replies": [] } ] }, { "id": 3014283, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-10-11T05:42:03.297000", "content": "I used a very simple model, local cv is ~0.7, online score is much lower. Is there anything missing that I did not noticeβ¦ I do not think I would be able to figure it out before the end of the competition β¦π
", "votes": 0, "replies": [ { "id": 3014303, "author_name": "Johnny Lee", "author_url": "", "post_date": "2024-10-11T06:20:11.640000", "content": "It's the same here. I'm considering the following:
\nTo be honest, they should make the only difference between train/test data the new star. Currently I do not have any idea how to optimize β¦ only submit & zeros
", "votes": 0, "replies": [] }, { "id": 3014493, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2024-10-11T09:27:08.810000", "content": "Just so you're aware: there are two new stars (so 4 stars total in the test set). You can verify this with an appropriate assert in a submission script.
", "votes": 4, "replies": [] }, { "id": 3014494, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2024-10-11T09:27:19.670000", "content": "@wuliaokaola Hello, Johnny. what do you mean 'The new star (=2)'? Can we probe the test dataset?
", "votes": 0, "replies": [] }, { "id": 3014558, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-10-11T10:54:03.837000", "content": "50% data for star0/1, 50% data for star2/3
", "votes": 1, "replies": [] }, { "id": 3023537, "author_name": "Faithful Chukwunwogor", "author_url": "", "post_date": "2024-10-20T16:56:48.133000", "content": "Hi, how did you finally resolve the zero score error?
", "votes": 0, "replies": [] } ] }, { "id": 3014462, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2024-10-11T09:10:06.263000", "content": "That's normal and I also meet the same situation. It's easy to train and predict the wavelengths, and I try finding the different pre- or proprocessing for test datasets.
", "votes": 0, "replies": [] }, { "id": 3015477, "author_name": "", "author_url": "", "post_date": "2024-10-12T13:22:52.197000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "3012480": "I struggled with 0 score for weeks. In my case the key difference between train and test was probably that the ingress and egress times (i.e. when the planet enters and exits) are different in test. So if you're making specific assumptions on when this happens (for example by removing a large amount of data around the possible ingress and egress), you can run into this.", "3012413": "It seems that my nn model gives a reasonable cv score (>0.55) across kfold, but the LB score is always 0. What could be the reason? Could it be that my model give a prediction too big because the distribution of test set is different? ", "3012438": "Try increasing sigma prediction. Uncertainty for test set predictions is much higher and caused people to have zero score", "3022135": "Also encountered this problem, would you share your idea on how to resolve it?", "3014283": "I used a very simple model, local cv is ~0.7, online score is much lower. Is there anything missing that I did not notice... I do not think I would be able to figure it out before the end of the competition ...π " } }