\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18688912%2F64666f6930414949f316816216af3df7%2Fscreenshot1.png?generation=1729329246828724&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18688912%2F157bcc97730cde07c41617582c32e4af%2Fscreenshot2.png?generation=1729329260413294&alt=media\")
In public notebooks I can see that sigma is just a pre-defined constant for all wavelengths:
= np.es_like(predictions_spectra) * \nor
\n = np.es_like(train_s) * \nHow these constants (0.000145 or 0.00016) were chosen?
If you predict only the mean, the best is to set a fixe sigma to have on average the maximum of wavelenghts in your +-sigma. More your mean is stable more you can reduce the sigma. But there is a limit if you predict a constant
", "rawMarkdown": "If you predict only the mean, the best is to set a fixe sigma to have on average the maximum of wavelenghts in your +-sigma. More your mean is stable more you can reduce the sigma. But there is a limit if you predict a constant", "votes": 5, "replies": [ { "id": 3022084, "postDate": "2024-10-19T08:51:04.900Z", "content": "how were you able to predict the mean so accurately. I hope this does not give away the important info.
\nI am trying sergei's ariel only correlation notebook to predict the mean, i did not use any model however.
\nTo predict the mean so accurately do i need to use parametrised models?
does this screenshot represents most of the mean estimation for the planets?
\nare there still planets whose mean would lie absolutely below or above the depth vs spectra curve?
I use another method than sergei but it's close in some way. I have around 1.6% variation on the model to predict the mean. But the star1 is more difficult. Now I'm thinking of a way to adapt the sigma. Because I have a model for the shape of the spectrum. I use sigma = 0.0001 for all stars and planet and my score is 0.555
\n
\n
Your prediction of the spectrum shape looks nice! Is it based on a parametric model? I tried some parametric models to predict the spectrum shape, but they couldn't generalize to the test set.
", "rawMarkdown": "Your prediction of the spectrum shape looks nice! Is it based on a parametric model? I tried some parametric models to predict the spectrum shape, but they couldn't generalize to the test set." }, { "id": 3022283, "postDate": "2024-10-19T12:40:52.137Z", "content": "I'm using a simple FNN.
", "rawMarkdown": "I'm using a simple FNN." }, { "id": 3022317, "postDate": "2024-10-19T13:02:00.063Z", "content": "May I ask what’s your RMSE using only the mean predictions? I am getting around 96-97 PPM for means alone but am struggling to make progress in the distribution across wavelengths.
", "rawMarkdown": "May I ask what’s your RMSE using only the mean predictions? I am getting around 96-97 PPM for means alone but am struggling to make progress in the distribution across wavelengths." }, { "id": 3022342, "postDate": "2024-10-19T13:22:02.640Z", "content": "RMSE*1e6: 97.54142863858668
", "rawMarkdown": "RMSE*1e6: 97.54142863858668" }, { "id": 3022465, "postDate": "2024-10-19T15:55:29.340Z", "content": "I'm also trying MLP (CV 0.578, RMSE 6.5e-5), yet also score 0 on LB, no idea where did I missed… :(
", "rawMarkdown": "I'm also trying MLP (CV 0.578, RMSE 6.5e-5), yet also score 0 on LB, no idea where did I missed... :(" }, { "id": 3022503, "postDate": "2024-10-19T16:21:39.660Z", "content": "Usually the 0 scores on LB are due to sigma overestimation and miscalculation of transit periods. Try looking at that, and think about the fact that there's 2 more stars in the test set.
", "rawMarkdown": "Usually the 0 scores on LB are due to sigma overestimation and miscalculation of transit periods. Try looking at that, and think about the fact that there's 2 more stars in the test set.", "votes": 3 }, { "id": 3022518, "postDate": "2024-10-19T16:40:03.907Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 3024075, "postDate": "2024-10-21T10:00:24.310Z", "content": "I am able to predict mean. GLL score for mean with std alone is 0.63 in given dataset, but i think i mine is not good at predicting fluctuations. When i add fluctuations score becomes 0.47.
", "rawMarkdown": "I am able to predict mean. GLL score for mean with std alone is 0.63 in given dataset, but i think i mine is not good at predicting fluctuations. When i add fluctuations score becomes 0.47." } ] } ] }, { "id": 3021819, "postDate": "2024-10-18T22:54:13.677Z", "content": "In public notebooks I can see that sigma is just a pre-defined constant for all wavelengths:
= np.es_like(predictions_spectra) * \nor
\n = np.es_like(train_s) * \nHow these constants (0.000145 or 0.00016) were chosen?
Most likely chosen to overfit the public leaderboard or training data
", "rawMarkdown": "Most likely chosen to overfit the public leaderboard or training data", "votes": 2, "replies": [ { "id": 3022258, "postDate": "2024-10-19T12:26:41.987Z", "content": "Probably not? If we use sergei's ariel only correlation notebook and calculate the max error over all the wavelengths of each planet, then the average of such a maximum error is about 0.00015.
", "rawMarkdown": "Probably not? If we use sergei's ariel only correlation notebook and calculate the max error over all the wavelengths of each planet, then the average of such a maximum error is about 0.00015.", "replies": [ { "id": 3022375, "postDate": "2024-10-19T13:51:47.260Z", "content": "The question asked was why did the notebooks use a constant sigma without any explanation or calculation.
\nAnd the most likely case is they simply picked the number that gave the best score.
\nWhat you described is overfitting. We cannot calculate the error during prediction, so we cannot determine the right sigma over the test set using this msthod. And there's a distribution shift in the test set, so a constant sigma is likely not ideal.
\nThe correct approach is to see if you can find a way to predict the sigma using variables available at the time of prediction. And then apply it to unlabeled data. Which the public notebooks do not do.
", "rawMarkdown": "The question asked was why did the notebooks use a constant sigma without any explanation or calculation.\n\nAnd the most likely case is they simply picked the number that gave the best score.\n\nWhat you described is overfitting. We cannot calculate the error during prediction, so we cannot determine the right sigma over the test set using this msthod. And there's a distribution shift in the test set, so a constant sigma is likely not ideal.\n\nThe correct approach is to see if you can find a way to predict the sigma using variables available at the time of prediction. And then apply it to unlabeled data. Which the public notebooks do not do.", "votes": 7 }, { "id": 3022385, "postDate": "2024-10-19T13:58:17.023Z", "content": "Thanks for your reply! I get it.
", "rawMarkdown": "Thanks for your reply! I get it." }, { "id": 3022553, "postDate": "2024-10-19T17:20:18.180Z", "content": "Yes, I took it from another public notebook blindly.
", "rawMarkdown": "Yes, I took it from another public notebook blindly.", "votes": 2 } ] } ] } ], "comments": [ { "id": 3021837, "author_name": "Félix Truong", "author_url": "", "post_date": "2024-10-18T23:55:07.190000", "content": "If you predict only the mean, the best is to set a fixe sigma to have on average the maximum of wavelenghts in your +-sigma. More your mean is stable more you can reduce the sigma. But there is a limit if you predict a constant
", "votes": 5, "replies": [ { "id": 3022084, "author_name": "highDopamine", "author_url": "", "post_date": "2024-10-19T08:51:04.900000", "content": "how were you able to predict the mean so accurately. I hope this does not give away the important info.
\nI am trying sergei's ariel only correlation notebook to predict the mean, i did not use any model however.
\nTo predict the mean so accurately do i need to use parametrised models?
does this screenshot represents most of the mean estimation for the planets?
\nare there still planets whose mean would lie absolutely below or above the depth vs spectra curve?
I use another method than sergei but it's close in some way. I have around 1.6% variation on the model to predict the mean. But the star1 is more difficult. Now I'm thinking of a way to adapt the sigma. Because I have a model for the shape of the spectrum. I use sigma = 0.0001 for all stars and planet and my score is 0.555
\n
\n
Your prediction of the spectrum shape looks nice! Is it based on a parametric model? I tried some parametric models to predict the spectrum shape, but they couldn't generalize to the test set.
", "votes": 0, "replies": [] }, { "id": 3022283, "author_name": "Félix Truong", "author_url": "", "post_date": "2024-10-19T12:40:52.137000", "content": "I'm using a simple FNN.
", "votes": 0, "replies": [] }, { "id": 3022317, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2024-10-19T13:02:00.063000", "content": "May I ask what’s your RMSE using only the mean predictions? I am getting around 96-97 PPM for means alone but am struggling to make progress in the distribution across wavelengths.
", "votes": 0, "replies": [] }, { "id": 3022342, "author_name": "Félix Truong", "author_url": "", "post_date": "2024-10-19T13:22:02.640000", "content": "RMSE*1e6: 97.54142863858668
", "votes": 0, "replies": [] }, { "id": 3022465, "author_name": "ZHANG XINGHAN", "author_url": "", "post_date": "2024-10-19T15:55:29.340000", "content": "I'm also trying MLP (CV 0.578, RMSE 6.5e-5), yet also score 0 on LB, no idea where did I missed… :(
", "votes": 0, "replies": [] }, { "id": 3022503, "author_name": "Andrei Zamfir", "author_url": "", "post_date": "2024-10-19T16:21:39.660000", "content": "Usually the 0 scores on LB are due to sigma overestimation and miscalculation of transit periods. Try looking at that, and think about the fact that there's 2 more stars in the test set.
", "votes": 3, "replies": [] }, { "id": 3022518, "author_name": "", "author_url": "", "post_date": "2024-10-19T16:40:03.907000", "content": "", "votes": 2, "replies": [] }, { "id": 3024075, "author_name": "Ajay Varghese", "author_url": "", "post_date": "2024-10-21T10:00:24.310000", "content": "I am able to predict mean. GLL score for mean with std alone is 0.63 in given dataset, but i think i mine is not good at predicting fluctuations. When i add fluctuations score becomes 0.47.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3021976, "author_name": "JungleBeastDS", "author_url": "", "post_date": "2024-10-19T05:53:19.090000", "content": "Most likely chosen to overfit the public leaderboard or training data
", "votes": 2, "replies": [ { "id": 3022258, "author_name": "Zhu Siqi", "author_url": "", "post_date": "2024-10-19T12:26:41.987000", "content": "Probably not? If we use sergei's ariel only correlation notebook and calculate the max error over all the wavelengths of each planet, then the average of such a maximum error is about 0.00015.
", "votes": 0, "replies": [ { "id": 3022375, "author_name": "JungleBeastDS", "author_url": "", "post_date": "2024-10-19T13:51:47.260000", "content": "The question asked was why did the notebooks use a constant sigma without any explanation or calculation.
\nAnd the most likely case is they simply picked the number that gave the best score.
\nWhat you described is overfitting. We cannot calculate the error during prediction, so we cannot determine the right sigma over the test set using this msthod. And there's a distribution shift in the test set, so a constant sigma is likely not ideal.
\nThe correct approach is to see if you can find a way to predict the sigma using variables available at the time of prediction. And then apply it to unlabeled data. Which the public notebooks do not do.
", "votes": 7, "replies": [] }, { "id": 3022385, "author_name": "Zhu Siqi", "author_url": "", "post_date": "2024-10-19T13:58:17.023000", "content": "Thanks for your reply! I get it.
", "votes": 0, "replies": [] }, { "id": 3022553, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2024-10-19T17:20:18.180000", "content": "Yes, I took it from another public notebook blindly.
", "votes": 2, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3021837": "If you predict only the mean, the best is to set a fixe sigma to have on average the maximum of wavelenghts in your +-sigma. More your mean is stable more you can reduce the sigma. But there is a limit if you predict a constant", "3021819": "In public notebooks I can see that `sigma` is just a pre-defined constant for all wavelengths:\n```\nsigmas = np.ones_like(predictions_spectra) * 0.000145\n```\nor\n```\ntrain_sigma = np.ones_like(train_s) * 0.00016\n```\n\nHow these constants (`0.000145` or `0.00016`) were chosen? ", "3021976": "Most likely chosen to overfit the public leaderboard or training data" } }