![\"score\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2Fd88037edd3819571c6cc55811a847e23%2Fscore.png?generation=1723665074565505&alt=media\")
In this competition, we submit a set of predicted targets \\(\\mu_{user}\\) and predicted uncertainties \\(\\sigma_{user}\\). The competition metric evaluates these predictions against the ground truth pixel level spectrum (\\(y\\)) using the Gaussian Log-likelihood (GLL) function.
\nAstronomers (and businesspeople, too) like uncertainty quantification: They not only want to see predictions, they want to know as well how confident we are with these predictions.
\nIf we submit the same \\(\\sigma_{user}\\) for all samples, the formula for the public leaderboard score can be simplified to
\n\\[\\large{score = \\frac {\\log \\sigma_{user}^2 + \\frac{MSE(y, \\mu_{user})}{\\sigma_{user}^2} + 9.16}{-23.03 + 9.16}}\\]
\n
This means:
\nA standard recipe to optimize this scoring function is: Predict \\(\\mu_{user}\\) with any regression model which optimizes MSE and then set \\(\\sigma_{user}\\) to the root mean squared error of the out-of-fold predictions. This is what I did in version 1 of my ADC24 Intro training and ADC24 Intro inference notebooks: The oof RMSE was 0.000293, I set sigma_pred=0.000293, the cv score was 0.259 and the public leaderboard score was 0.114.
Where does this huge cv–lb difference come from? The standard recipe doesn't take into account that train and test distributions are different. In the training dataset all planets belong to two host stars numbered 0 and 1. The test dataset is more diverse: In addition to stars 0 and 1, there is at least one other (previously unseen) star. Adapting the standard recipe to this situation, I'd propose the following:
\n\n\nFor known stars set \\(\\sigma_{user}\\) to the out-of-fold rmse, and for unknown stars set it to a higher value. Perhaps use the out-of-fold rmse of a GroupKFold to estimate \\(\\sigma_{user}\\) for the unknown stars.
\n
EDIT: The recipe is only valid as a baseline. A more sophisticated model will of course predict individual sigmas for every planet and wavelength.
", "messages": [ { "id": 2959341, "postDate": "2024-08-14T19:52:52.720Z", "content": "In this competition, we submit a set of predicted targets \\(\\mu_{user}\\) and predicted uncertainties \\(\\sigma_{user}\\). The competition metric evaluates these predictions against the ground truth pixel level spectrum (\\(y\\)) using the Gaussian Log-likelihood (GLL) function.
\nAstronomers (and businesspeople, too) like uncertainty quantification: They not only want to see predictions, they want to know as well how confident we are with these predictions.
\nIf we submit the same \\(\\sigma_{user}\\) for all samples, the formula for the public leaderboard score can be simplified to
\n\\[\\large{score = \\frac {\\log \\sigma_{user}^2 + \\frac{MSE(y, \\mu_{user})}{\\sigma_{user}^2} + 9.16}{-23.03 + 9.16}}\\]
\n
This means:
\nA standard recipe to optimize this scoring function is: Predict \\(\\mu_{user}\\) with any regression model which optimizes MSE and then set \\(\\sigma_{user}\\) to the root mean squared error of the out-of-fold predictions. This is what I did in version 1 of my ADC24 Intro training and ADC24 Intro inference notebooks: The oof RMSE was 0.000293, I set sigma_pred=0.000293, the cv score was 0.259 and the public leaderboard score was 0.114.
Where does this huge cv–lb difference come from? The standard recipe doesn't take into account that train and test distributions are different. In the training dataset all planets belong to two host stars numbered 0 and 1. The test dataset is more diverse: In addition to stars 0 and 1, there is at least one other (previously unseen) star. Adapting the standard recipe to this situation, I'd propose the following:
\n\n\nFor known stars set \\(\\sigma_{user}\\) to the out-of-fold rmse, and for unknown stars set it to a higher value. Perhaps use the out-of-fold rmse of a GroupKFold to estimate \\(\\sigma_{user}\\) for the unknown stars.
\n
EDIT: The recipe is only valid as a baseline. A more sophisticated model will of course predict individual sigmas for every planet and wavelength.
", "rawMarkdown": "In this competition, we submit a set of predicted targets \\\\(\\mu_{user}\\\\) and predicted uncertainties \\\\(\\sigma_{user}\\\\). The competition metric evaluates these predictions against the ground truth pixel level spectrum (\\\\(y\\\\)) using the Gaussian Log-likelihood (GLL) function.\n\nAstronomers (and businesspeople, too) like uncertainty quantification: They not only want to see predictions, they want to know as well how confident we are with these predictions. \n\nIf we submit the same \\\\(\\sigma_{user}\\\\) for all samples, the formula for the public leaderboard score can be simplified to\n\n\\\\[\\large{score = \\frac {\\log \\sigma_{user}^2 + \\frac{MSE(y, \\mu_{user})}{\\sigma_{user}^2} + 9.16}{-23.03 + 9.16}}\\\\]\n\n\n\nThis means:\n1. The score is a (linearly transformed) sum of two terms.\n2. Whatever the value of \\\\(\\sigma_{user}\\\\) is, we want to optimize the mean squared error (MSE) of the \\\\(\\mu_{user}\\\\) predictions as usual. This is a standard regression task.\n3. After our regression model has predicted \\\\(\\mu_{user}\\\\) for every sample, we may choose a \\\\(\\sigma_{user}\\\\). If we choose a very high \\\\(\\sigma_{user}\\\\), the first term of the formula (\\\\(\\log \\sigma_{user}^2\\\\)) dominates. If we choose a low (positive) \\\\(\\sigma_{user}\\\\), the second term dominates. The optimum is somewhere in between (see diagram above).\n4. The diagram furthermore shows that a low \\\\(\\sigma_{user}\\\\) is punished more than one which is too high.\n4. The optimum \\\\(\\sigma_{user}\\\\), which maximizes the score for a given MSE, is \\\\(\\sigma_{user} = \\sqrt{MSE}\\\\). (To see this, compute the derivative with respect to \\\\(\\sigma_{user}^2\\\\) and set it to zero.)\n\nA standard recipe to optimize this scoring function is: Predict \\\\(\\mu_{user}\\\\) with any regression model which optimizes MSE and then set \\\\(\\sigma_{user}\\\\) to the root mean squared error of the out-of-fold predictions. This is what I did in version 1 of my [ADC24 Intro training](https://www.kaggle.com/code/ambrosm/adc24-intro-training) and [ADC24 Intro inference](https://www.kaggle.com/code/ambrosm/adc24-intro-inference) notebooks: The oof RMSE was 0.000293, I set `sigma_pred=0.000293`, the cv score was 0.259 and the public leaderboard score was 0.114.\n\nWhere does this huge cv–lb difference come from? The standard recipe doesn't take into account that train and test distributions are different. In the training dataset all planets belong to two host stars numbered 0 and 1. The test dataset is more diverse: In addition to stars 0 and 1, there is at least one other (previously unseen) star. Adapting the standard recipe to this situation, I'd propose the following:\n\n> For known stars set \\\\(\\sigma_{user}\\\\) to the out-of-fold rmse, and for unknown stars set it to a higher value. Perhaps use the out-of-fold rmse of a GroupKFold to estimate \\\\(\\sigma_{user}\\\\) for the unknown stars.\n\nEDIT: The recipe is only valid as a baseline. A more sophisticated model will of course predict individual sigmas for every planet and wavelength.\n", "votes": 61 }, { "id": 2960975, "postDate": "2024-08-16T10:18:17.833Z", "content": "@ambrosm Thanks you for publishing comprehensive guide.
\nI think I found mistake on your calculation.
\n\nFor L_ref is defined using the mean and variance of the training dataset as its prediction for all instances.
\n
I calculated L_ref in your customized GLL, it should be -11.72 not -9.16.
\nWhen using this value, the CV of your score 0.259 becomes 0.091 which is closer to your LB score.
\nFeel free to point out if I make mistakes.
polars pl\n\n ():\n var = sigmas**\n mse = (preds - targets) ** \n score = np.log(var) + mse / var\n score.mean()\n\n\n ():\n gll_simple = score * (- + ) - \n score = (gll_simple - gll_trivial) / (gll_opt - gll_trivial)\n score\n\n\ntargets = (\n pl.read_csv()\n .drop()\n .to_numpy()\n)\nx_mean = targets.mean(axis=, keepdims=)\nx_std = targets.std(axis=, keepdims=)\n\n\n(calc_simple_metric(x_mean, x_std, targets)) \n\n\n(calc_simple_metric(targets, , targets)) \n\ncorrect_score() \nHi @tatamikenn, thank you for verifying my calculation. I neglected to mention that there are three different values for L_ref: These are L_ref_train, L_ref_publictest, L_ref_privatetest. The difference comes from evaluating mean and variance of the training dataset against the true labels of the training dataset, the public test set or the private test set, respectively.
\n@ambrosm How do you calculated L_re_publictest estimation -9.16 ?
", "rawMarkdown": "@ambrosm How do you calculated L_re_publictest estimation **-9.16** ?", "votes": 1 } ] } ] }, { "id": 2959362, "postDate": "2024-08-14T20:14:09.733Z", "content": "Great rundown of the metric! I am quite excited about this competition and digging a bit through the data now to get started.
\n\n\nIn addition to stars 0 and 1, there is at least one other (previously unseen) star. Adapting the standard recipe to this situation, I'd propose the following
\n
This is the first time that I read it like this. Was it confirmed that test includes star 0 and star 1 from training? It sounded like it contains \"only\" a new solar system.
", "rawMarkdown": "Great rundown of the metric! I am quite excited about this competition and digging a bit through the data now to get started.\n\n> In addition to stars 0 and 1, there is at least one other (previously unseen) star. Adapting the standard recipe to this situation, I'd propose the following\n\nThis is the first time that I read it like this. Was it confirmed that test includes star 0 and star 1 from training? It sounded like it contains \"only\" a new solar system.\n", "votes": 5, "replies": [ { "id": 2959687, "postDate": "2024-08-15T07:19:07.383Z", "content": "@ilu000 Host conformed about new star for 327 exoplanets, so other 473 need to be belongs to star 0 / 1 / both
\n\n", "rawMarkdown": "@ilu000 [Host conformed](https://www.kaggle.com/competitions/ariel-data-challenge-2024/discussion/524287#2950440) about new star for 327 exoplanets, so other 473 need to be belongs to star 0 / 1 / both\n\n> Its having 2 solar systems => first with 346 exoplanets and second with 327 exoplanets, is remaining 327 exoplanets (test) belongs to same solar systems or new solar system?\nThe 327 belongs to a new planetary system (or new host star)\n800 - 327 => 473 exoplanets belongs to star 0 / 1 / both ", "votes": 3 }, { "id": 2959789, "postDate": "2024-08-15T09:53:49.737Z", "content": "Its having 2 solar systems => first with 346 exoplanets and second with 327 exoplanets, is remaining 327 exoplanets (test) belongs to same solar systems or new solar system?
\n
\n The 327 belongs to a new planetary system (or new host star)
\n 800 - 327 => 473 exoplanets belongs to star 0 / 1 / both
Hi @ilu000, the test set includes both previously seen and previously unseen stars.
", "rawMarkdown": "Hi @ilu000, the test set includes both previously seen and previously unseen stars.", "votes": 8 } ] }, { "id": 2994368, "postDate": "2024-09-20T20:39:00.970Z", "content": "Thanks for the clear, simple presentation!
\nOne comment: the \\(L_{ideal}\\) is based on \\(MSE/\\sigma^2=0\\), fair enough. But in practice this term will be at least \\((\\sigma_{sim}/10^{-5})^2\\), where \\(\\sigma_{sim}\\) is the error term in the simulated data. So, for \\(\\sigma_{sim}=10^{-5}\\) the best score is about 0.93
@ambrosm Thanks you for publishing comprehensive guide.
\nI think I found mistake on your calculation.
\n\nFor L_ref is defined using the mean and variance of the training dataset as its prediction for all instances.
\n
I calculated L_ref in your customized GLL, it should be -11.72 not -9.16.
\nWhen using this value, the CV of your score 0.259 becomes 0.091 which is closer to your LB score.
\nFeel free to point out if I make mistakes.
polars pl\n\n ():\n var = sigmas**\n mse = (preds - targets) ** \n score = np.log(var) + mse / var\n score.mean()\n\n\n ():\n gll_simple = score * (- + ) - \n score = (gll_simple - gll_trivial) / (gll_opt - gll_trivial)\n score\n\n\ntargets = (\n pl.read_csv()\n .drop()\n .to_numpy()\n)\nx_mean = targets.mean(axis=, keepdims=)\nx_std = targets.std(axis=, keepdims=)\n\n\n(calc_simple_metric(x_mean, x_std, targets)) \n\n\n(calc_simple_metric(targets, , targets)) \n\ncorrect_score() \nHi @tatamikenn, thank you for verifying my calculation. I neglected to mention that there are three different values for L_ref: These are L_ref_train, L_ref_publictest, L_ref_privatetest. The difference comes from evaluating mean and variance of the training dataset against the true labels of the training dataset, the public test set or the private test set, respectively.
\n@ambrosm How do you calculated L_re_publictest estimation -9.16 ?
", "votes": 1, "replies": [] } ] } ] }, { "id": 2959362, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2024-08-14T20:14:09.733000", "content": "Great rundown of the metric! I am quite excited about this competition and digging a bit through the data now to get started.
\n\n\nIn addition to stars 0 and 1, there is at least one other (previously unseen) star. Adapting the standard recipe to this situation, I'd propose the following
\n
This is the first time that I read it like this. Was it confirmed that test includes star 0 and star 1 from training? It sounded like it contains \"only\" a new solar system.
", "votes": 5, "replies": [ { "id": 2959687, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2024-08-15T07:19:07.383000", "content": "@ilu000 Host conformed about new star for 327 exoplanets, so other 473 need to be belongs to star 0 / 1 / both
\n\n", "votes": 3, "replies": [] }, { "id": 2959789, "author_name": "AmbrosM", "author_url": "", "post_date": "2024-08-15T09:53:49.737000", "content": "Its having 2 solar systems => first with 346 exoplanets and second with 327 exoplanets, is remaining 327 exoplanets (test) belongs to same solar systems or new solar system?
\n
\n The 327 belongs to a new planetary system (or new host star)
\n 800 - 327 => 473 exoplanets belongs to star 0 / 1 / both
Hi @ilu000, the test set includes both previously seen and previously unseen stars.
", "votes": 8, "replies": [] } ] }, { "id": 2994368, "author_name": "Daniel Dewey", "author_url": "", "post_date": "2024-09-20T20:39:00.970000", "content": "Thanks for the clear, simple presentation!
\nOne comment: the \\(L_{ideal}\\) is based on \\(MSE/\\sigma^2=0\\), fair enough. But in practice this term will be at least \\((\\sigma_{sim}/10^{-5})^2\\), where \\(\\sigma_{sim}\\) is the error term in the simulated data. So, for \\(\\sigma_{sim}=10^{-5}\\) the best score is about 0.93