![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fd7001b68bae626d71e9e16da10f86f74%2Frisk_graph.jpeg?generation=1716355019829983&alt=media\")
Hi. Hope you and your fellow Sisyphus are doing well with carrying the rock.
\nI'll show some evidence of this competition's extreme setup.
\nsome target columns are significantly risky.
\nIf one single outlier sample could cause significant drop on private LB (~-12 with single sample).
I plotted the risk of all 368 target columns except trivial columns with σ=0.
\nThe risk is calculated by
$$
\n\\text{risk} := \\max(y) / \\sigma(y)
\n$$
The first and 2nd picture shows some columns has significantly distant outliers (up to ~2500 σ).
\nThis could cause your model significantly drops in private LB.
\nThe 3rd picture depicts this outlier case.
\nSuppose if your model predict single outlier in private test set with ptend_q0002_27 = 0 (because 99.9999% has <= 0 values), whose actual value is 2523σ, you will get current score - 12.24 in private LB.
So hope your model can handle these outliers well.
\n(I also hope host's already removed these miserable outliers from test set).
Good luck!
\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fe359364d9949eb7c67a8c1e42496dd6b%2FScreenshot%202024-05-22%20at%2014.16.39.png?generation=1716355036821574&alt=media\")
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F30d088e4934ee91f6b47c4862217b3de%2FScreenshot%202024-05-22%20at%2014.16.47.png?generation=1716355049231387&alt=media\")
Hi. Hope you and your fellow Sisyphus are doing well with carrying the rock.
\nI'll show some evidence of this competition's extreme setup.
\nsome target columns are significantly risky.
\nIf one single outlier sample could cause significant drop on private LB (~-12 with single sample).
I plotted the risk of all 368 target columns except trivial columns with σ=0.
\nThe risk is calculated by
$$
\n\\text{risk} := \\max(y) / \\sigma(y)
\n$$
The first and 2nd picture shows some columns has significantly distant outliers (up to ~2500 σ).
\nThis could cause your model significantly drops in private LB.
\nThe 3rd picture depicts this outlier case.
\nSuppose if your model predict single outlier in private test set with ptend_q0002_27 = 0 (because 99.9999% has <= 0 values), whose actual value is 2523σ, you will get current score - 12.24 in private LB.
So hope your model can handle these outliers well.
\n(I also hope host's already removed these miserable outliers from test set).
Good luck!
\n
Hello, thank you for providing this detailed analysis of the dataset and sharing your concerns. If it helps, the private leaderboard score is only half a percent different from the public leaderboard score for the u-net benchmark submission.
", "rawMarkdown": "Hello, thank you for providing this detailed analysis of the dataset and sharing your concerns. If it helps, the private leaderboard score is only half a percent different from the public leaderboard score for the u-net benchmark submission.", "votes": 14, "replies": [ { "id": 2834437, "postDate": "2024-05-24T18:19:40.280Z", "content": "That's good to know but I don't think it completely negates the risk of getting negative score.
", "rawMarkdown": "That's good to know but I don't think it completely negates the risk of getting negative score.", "votes": 2 }, { "id": 2834672, "postDate": "2024-05-24T22:41:18.327Z", "content": "@jerrylin96 Thanks. It somehow helped.
\nHowever, as Gunes pointed out, there left some risk of our (i.e. Kaggle community's) best performant model would be significantly affected by tiny amount of extreme outliers.
\nIs it possible to consider to take further actions like listed in [1]?
\nFor example, if we have the baseline model's prediction for all test set (i.e. submission.csv), we can test our model's safeness by checking if our model's predictions are not significantly apart from this prediction.
[1] https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2832998
", "rawMarkdown": "@jerrylin96 Thanks. It somehow helped.\n\nHowever, as Gunes pointed out, there left some risk of our (i.e. Kaggle community's) best performant model would be significantly affected by tiny amount of extreme outliers.\n\nIs it possible to consider to take further actions like listed in [1]?\nFor example, if we have the baseline model's prediction for all test set (i.e. `submission.csv`), we can test our model's safeness by checking if our model's predictions are not significantly apart from this prediction.\n\n## Reference\n\n[1] https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2832998", "votes": 2, "replies": [ { "id": 2835414, "postDate": "2024-05-25T10:03:05.023Z", "content": "Also can we get details about the experimental setup (train/test size and etc.) of the u-net model?
", "rawMarkdown": "Also can we get details about the experimental setup (train/test size and etc.) of the u-net model?", "votes": 1 } ] } ] }, { "id": 2832998, "postDate": "2024-05-24T02:58:01.050Z", "content": "I think generally machine learning models under current competition metrics (R2) is prone to some kind of extreme outliers (i.e. 2523σ).
\nIf this kind of extreme outliers are in private test set, it possibly causes catastrophic shakeup in the end of the competition.
Hopefully, we believe some safeguard may be helpful for all of us. like,
\nc.f.) I also posted mathematical explanation for instability in R2 metrics[4].
\n\nHow hosts and Kaggle team think of this risk?
\nThanks in advance.
\namazing study
\nsuggestion:
\nmacro-avg(r2-score) -> macro-avg(clip(r2-score,0,1))
this way even if the participant doesn't handle the outlier, he is losing ~0.003 per-column compared to the perfectly predicted column (it hurts, but better than to have -10 score in private)
", "rawMarkdown": "amazing study\n\nsuggestion:\nmacro-avg(r2-score) -> macro-avg(clip(r2-score,0,1))\n\nthis way even if the participant doesn't handle the outlier, he is losing ~0.003 per-column compared to the perfectly predicted column (it hurts, but better than to have -10 score in private)", "votes": 6, "replies": [ { "id": 2829800, "postDate": "2024-05-22T20:05:26.670Z", "content": "I will suggest a fix of my own then.
\nDefinition of r2 (I will use Wikipedia notation):
\nr2 = 1-SS(res)/SS(tot)
The problem: since SS(tot) is calculated only for the test set, there can be a column with very small values for the private test set, leading to SS(tot) very small, leading to a negative explosion of r2 for this column.
\nFix: Use SS(tot) that is calculated for all the data. So instead of SS(tot)(test):
\n(n(#samples in test)/N(#all samples))*SS(tot)(all samples)
\nBasically, ensure that SS(tot) has reasonable values. Actually, the values should just by n since the values are normalized by their SD. So it can be reduced to:
\nr2 = 1-SS(res)/n(#samples in test)
\nI think? Well, something along those lines. The general idea should be correct.
First of all, we should claim the relationship between $R^2$ metrics and standard deviation of given dataset. By definition,
\n$$
\n\\begin{align*}
\nR^2 &= 1 - \\frac{\\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2}{\\frac{1}{N} \\sum_i (y_i - \\bar{y})^2} \\\\
\n&=1 - \\frac{\\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2}{\\sigma_t^2} \\\\
\n&=1 - \\frac{1}{N} \\sum_i (y_i/\\sigma_t - \\hat{y}_i / \\sigma_t)^2 \\\\
\n\\end{align*}
\n$$
That's why I normalized all data with standard deviation of all train data. And the above metric should be affected by outlier even if we use S(tot)_train instead of S(tot)_test i.e. sample with y_i=2523σ busts R2 score.
", "rawMarkdown": "First of all, we should claim the relationship between $R^2$ metrics and standard deviation of given dataset. By definition,\n\n$$\n\\begin{align\\*}\nR^2 &= 1 - \\frac{\\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2}{\\frac{1}{N} \\sum_i (y_i - \\bar{y})^2} \\\\\\\\\n&=1 - \\frac{\\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2}{\\sigma_t^2} \\\\\\\\\n&=1 - \\frac{1}{N} \\sum_i (y_i/\\sigma_t - \\hat{y}_i / \\sigma_t)^2 \\\\\\\\\n\\end{align\\*}\n$$\n\nThat's why I normalized all data with standard deviation of all train data. And the above metric **should be affected by outlier even if we use S(tot)_train instead of S(tot)_test** i.e. **sample with y_i=2523σ busts R2 score**.", "votes": 1 }, { "id": 2829931, "postDate": "2024-05-22T22:34:36.333Z", "content": "Besides, its unclear that host & Kaggle team admit changing competition metrics by its design in the middle of the competition (because it could cause biased merit for some teams and not for others etc.).
\nSo, my current idea is asking host that providing us some measure of safeness without changing metrics.
\n\nMy other thought was to ask host to publish their unet baseline build for this comp and ask clarification for it surely have decent score on private LB. This would be a benchmark of safeness.
\n
https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2828609
", "rawMarkdown": "Besides, its unclear that host & Kaggle team admit changing competition metrics by its design in the middle of the competition (because it could cause biased merit for some teams and not for others etc.).\nSo, my current idea is **asking host that providing us some measure of safeness without changing metrics**.\n\n> My other thought was to ask host to publish their unet baseline build for this comp and ask clarification for it surely have decent score on private LB. This would be a benchmark of safeness.\n\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2828609", "votes": 4 } ] } ] }, { "id": 2829050, "postDate": "2024-05-22T11:38:15.190Z", "content": "Let's calculate how extreme the resolution this competition requires to our model.
\nThis can be done with maximum value without normalizing with std deviation.
\nThe below table is statistics of target features without normalizing with std deviation.
\nSo, 2523σ in ptend_q0002_27 corresponds with 6.7934e-9 [kg/kg] in actual humidity unit.
\nHow this amount is difficult to predict?
Suppose density of dry air is 1 [kg/m^3], this amount corresponds with 6.7934 μg of vapor within 1 m^3.
\nI don't know how difficult this is (I appreciate kind person with sufficient domain knowledge suggest us).
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Ff2f917e5d3f55127ced497aec5c6d844%2FScreenshot%202024-05-22%20at%2020.30.37.png?generation=1716377449594314&alt=media\")
Difficulty vs Risk the targets
\nThe below picture is plots of R2 score of one of my best models, and the risk.
\nIt depicts most of peaks in high risk actually shows difficulty though some are not (e.g. ptend_q0002_27).
\nThe latter case might be the model predict according to input distribution rather than fair function approximation.
\nThis also shows the risky columns are actually difficult to predict.
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F8297a00fe5449f6df99578eca27472d8%2Fr2_vs_risk.jpeg?generation=1716380351091679&alt=media\")
@tatamikenn , this is an AMAZING plot. Thank you very much for sharing this with us! I am glad to see that your R2 curves look exactly like mine, although your ptend_u and ptend_v are much higher than mines. There is probably some trick to predict that ones, that only the people in the first places noticed, right? 😉
", "rawMarkdown": "@tatamikenn , this is an AMAZING plot. Thank you very much for sharing this with us! I am glad to see that your R2 curves look exactly like mine, although your ptend_u and ptend_v are much higher than mines. There is probably some trick to predict that ones, that only the people in the first places noticed, right? 😉" }, { "id": 2829121, "postDate": "2024-05-22T12:39:11.047Z", "content": "Actually, I couldn't have fond any kind of tricks in my models.
\nThough digging this deep could find some important insights about this competition.
So, you don't have any columns with negative (and 0 after replacement) R2? Impressive!
", "rawMarkdown": "So, you don't have any columns with negative (and 0 after replacement) R2? Impressive!" }, { "id": 2830463, "postDate": "2024-05-23T07:52:26.330Z", "content": "Yep, but I have no idea on the test set.
", "rawMarkdown": "Yep, but I have no idea on the test set.", "votes": 2 }, { "id": 2840664, "postDate": "2024-05-28T07:35:49.957Z", "content": "
\nEdit: I had bugs
Also thanks for the great visualizations, I've integrated them into my training loop!
", "rawMarkdown": "~~Just to clarify, the beginnings (after submission mask) of q1-q3 are predictable (positive r2) by model alone? (discounting manual post-processing like the q2 state replacement trick)~~\nEdit: I had bugs\n\nAlso thanks for the great visualizations, I've integrated them into my training loop!" } ] } ] }, { "id": 2828467, "postDate": "2024-05-22T06:04:49.037Z", "content": "I created my validation scheme based on this fact. Some of the targets are not stable between different folds. I do a similar simulation on 625k sized splits and found that some of the folds can get negative r2 score on q0001 12 13 14, q0002 27 28, q0003 12 13 14 targets. I have no idea what will happen if test set has those outliers.
", "rawMarkdown": "I created my validation scheme based on this fact. Some of the targets are not stable between different folds. I do a similar simulation on 625k sized splits and found that some of the folds can get negative r2 score on q0001 12 13 14, q0002 27 28, q0003 12 13 14 targets. I have no idea what will happen if test set has those outliers.", "votes": 2, "replies": [ { "id": 2828477, "postDate": "2024-05-22T06:19:39.457Z", "content": "Before posting this, I debated myself whether it was better to keep this fact to myself and find a clever way to dodge the issue, or to point it out to the organizers and hosts and ask for fair treatment. In the end, I realized that the easiest path was to confront everyone with this fact and share the suffering.
", "rawMarkdown": "Before posting this, I debated myself whether it was better to keep this fact to myself and find a clever way to dodge the issue, or to point it out to the organizers and hosts and ask for fair treatment. In the end, I realized that the easiest path was to confront everyone with this fact and share the suffering.", "votes": 10, "replies": [ { "id": 2828535, "postDate": "2024-05-22T06:47:56.717Z", "content": "So basically… the person/team that builds the model that is more robust to outliers will win?
", "rawMarkdown": "So basically... the person/team that builds the model that is more robust to outliers will win?" }, { "id": 2828557, "postDate": "2024-05-22T07:00:22.040Z", "content": "Thanks for sharing. I think keeping information like this to yourself doesn't worth it. It may still backfire at the end since the metric is not stable. Changing metric to weighted macro average r2 score would be fine for this competition imo.
", "rawMarkdown": "Thanks for sharing. I think keeping information like this to yourself doesn't worth it. It may still backfire at the end since the metric is not stable. Changing metric to weighted macro average r2 score would be fine for this competition imo." }, { "id": 2828584, "postDate": "2024-05-22T07:10:11.527Z", "content": "\n\nChanging metric to weighted macro average r2 score would be fine for this competition imo.
\n
Yeah. I think R2 (normalizing with unbounded sigma) as it is too extreme for this competition. At least custom metrics with normalizing clipped weight (sigma is clipped with some tiny value) should have been better?
", "rawMarkdown": "> Changing metric to weighted macro average r2 score would be fine for this competition imo.\n\nYeah. I think R2 (normalizing with unbounded sigma) as it is too extreme for this competition. At least custom metrics with normalizing clipped weight (sigma is clipped with some tiny value) should have been better?" }, { "id": 2828590, "postDate": "2024-05-22T07:12:04.380Z", "content": "\n\nSo basically… the person/team that builds the model that is more robust to outliers will win?
\n
Possibly. But it depends on the test set. If host already removed these outliers, winners in public LB highly possibly wins.
", "rawMarkdown": "> So basically… the person/team that builds the model that is more robust to outliers will win?\n\nPossibly. But it depends on the test set. If host already removed these outliers, winners in public LB highly possibly wins.", "votes": 2 }, { "id": 2828609, "postDate": "2024-05-22T07:19:45.567Z", "content": "My other thought was to ask host to publish their unet baseline build for this comp and ask clarification for it surely have decent score on private LB. This would be a benchmark of safeness.
", "rawMarkdown": "My other thought was to ask host to publish their unet baseline build for this comp and ask clarification for it surely have decent score on private LB. This would be a benchmark of safeness.", "votes": 1 } ] } ] }, { "id": 2881583, "postDate": "2024-06-20T19:43:53.260Z", "content": "Cool plot, what tool did you use to generate it? Meaning, did you just use Pandas, or something more sophisticated like Apache Beam? Also, what kind of hardware did you use? Some times I run out of memory trying to analyze the whole dataset.
", "rawMarkdown": "Cool plot, what tool did you use to generate it? Meaning, did you just use Pandas, or something more sophisticated like Apache Beam? Also, what kind of hardware did you use? Some times I run out of memory trying to analyze the whole dataset.", "replies": [ { "id": 2882300, "postDate": "2024-06-21T09:27:22.100Z", "content": "I used polars on 128 GB RAM PC.
", "rawMarkdown": "I used polars on 128 GB RAM PC." } ] } ], "comments": [ { "id": 2834423, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-05-24T18:09:36.983000", "content": "Hello, thank you for providing this detailed analysis of the dataset and sharing your concerns. If it helps, the private leaderboard score is only half a percent different from the public leaderboard score for the u-net benchmark submission.
", "votes": 14, "replies": [ { "id": 2834437, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-05-24T18:19:40.280000", "content": "That's good to know but I don't think it completely negates the risk of getting negative score.
", "votes": 2, "replies": [] }, { "id": 2834672, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-24T22:41:18.327000", "content": "@jerrylin96 Thanks. It somehow helped.
\nHowever, as Gunes pointed out, there left some risk of our (i.e. Kaggle community's) best performant model would be significantly affected by tiny amount of extreme outliers.
\nIs it possible to consider to take further actions like listed in [1]?
\nFor example, if we have the baseline model's prediction for all test set (i.e. submission.csv), we can test our model's safeness by checking if our model's predictions are not significantly apart from this prediction.
[1] https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2832998
", "votes": 2, "replies": [ { "id": 2835414, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-05-25T10:03:05.023000", "content": "Also can we get details about the experimental setup (train/test size and etc.) of the u-net model?
", "votes": 1, "replies": [] } ] } ] }, { "id": 2832998, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-24T02:58:01.050000", "content": "I think generally machine learning models under current competition metrics (R2) is prone to some kind of extreme outliers (i.e. 2523σ).
\nIf this kind of extreme outliers are in private test set, it possibly causes catastrophic shakeup in the end of the competition.
Hopefully, we believe some safeguard may be helpful for all of us. like,
\nc.f.) I also posted mathematical explanation for instability in R2 metrics[4].
\n\nHow hosts and Kaggle team think of this risk?
\nThanks in advance.
\namazing study
\nsuggestion:
\nmacro-avg(r2-score) -> macro-avg(clip(r2-score,0,1))
this way even if the participant doesn't handle the outlier, he is losing ~0.003 per-column compared to the perfectly predicted column (it hurts, but better than to have -10 score in private)
", "votes": 6, "replies": [ { "id": 2829800, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-22T20:05:26.670000", "content": "I will suggest a fix of my own then.
\nDefinition of r2 (I will use Wikipedia notation):
\nr2 = 1-SS(res)/SS(tot)
The problem: since SS(tot) is calculated only for the test set, there can be a column with very small values for the private test set, leading to SS(tot) very small, leading to a negative explosion of r2 for this column.
\nFix: Use SS(tot) that is calculated for all the data. So instead of SS(tot)(test):
\n(n(#samples in test)/N(#all samples))*SS(tot)(all samples)
\nBasically, ensure that SS(tot) has reasonable values. Actually, the values should just by n since the values are normalized by their SD. So it can be reduced to:
\nr2 = 1-SS(res)/n(#samples in test)
\nI think? Well, something along those lines. The general idea should be correct.
First of all, we should claim the relationship between $R^2$ metrics and standard deviation of given dataset. By definition,
\n$$
\n\\begin{align*}
\nR^2 &= 1 - \\frac{\\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2}{\\frac{1}{N} \\sum_i (y_i - \\bar{y})^2} \\\\
\n&=1 - \\frac{\\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2}{\\sigma_t^2} \\\\
\n&=1 - \\frac{1}{N} \\sum_i (y_i/\\sigma_t - \\hat{y}_i / \\sigma_t)^2 \\\\
\n\\end{align*}
\n$$
That's why I normalized all data with standard deviation of all train data. And the above metric should be affected by outlier even if we use S(tot)_train instead of S(tot)_test i.e. sample with y_i=2523σ busts R2 score.
", "votes": 1, "replies": [] }, { "id": 2829931, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-22T22:34:36.333000", "content": "Besides, its unclear that host & Kaggle team admit changing competition metrics by its design in the middle of the competition (because it could cause biased merit for some teams and not for others etc.).
\nSo, my current idea is asking host that providing us some measure of safeness without changing metrics.
\n\nMy other thought was to ask host to publish their unet baseline build for this comp and ask clarification for it surely have decent score on private LB. This would be a benchmark of safeness.
\n
https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2828609
", "votes": 4, "replies": [] } ] } ] }, { "id": 2829050, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-22T11:38:15.190000", "content": "Let's calculate how extreme the resolution this competition requires to our model.
\nThis can be done with maximum value without normalizing with std deviation.
\nThe below table is statistics of target features without normalizing with std deviation.
\nSo, 2523σ in ptend_q0002_27 corresponds with 6.7934e-9 [kg/kg] in actual humidity unit.
\nHow this amount is difficult to predict?
Suppose density of dry air is 1 [kg/m^3], this amount corresponds with 6.7934 μg of vapor within 1 m^3.
\nI don't know how difficult this is (I appreciate kind person with sufficient domain knowledge suggest us).
Difficulty vs Risk the targets
\nThe below picture is plots of R2 score of one of my best models, and the risk.
\nIt depicts most of peaks in high risk actually shows difficulty though some are not (e.g. ptend_q0002_27).
\nThe latter case might be the model predict according to input distribution rather than fair function approximation.
\nThis also shows the risky columns are actually difficult to predict.
@tatamikenn , this is an AMAZING plot. Thank you very much for sharing this with us! I am glad to see that your R2 curves look exactly like mine, although your ptend_u and ptend_v are much higher than mines. There is probably some trick to predict that ones, that only the people in the first places noticed, right? 😉
", "votes": 0, "replies": [] }, { "id": 2829121, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-22T12:39:11.047000", "content": "Actually, I couldn't have fond any kind of tricks in my models.
\nThough digging this deep could find some important insights about this competition.
So, you don't have any columns with negative (and 0 after replacement) R2? Impressive!
", "votes": 0, "replies": [] }, { "id": 2830463, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-23T07:52:26.330000", "content": "Yep, but I have no idea on the test set.
", "votes": 2, "replies": [] }, { "id": 2840664, "author_name": "sroger", "author_url": "", "post_date": "2024-05-28T07:35:49.957000", "content": "
\nEdit: I had bugs
Also thanks for the great visualizations, I've integrated them into my training loop!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2828467, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-05-22T06:04:49.037000", "content": "I created my validation scheme based on this fact. Some of the targets are not stable between different folds. I do a similar simulation on 625k sized splits and found that some of the folds can get negative r2 score on q0001 12 13 14, q0002 27 28, q0003 12 13 14 targets. I have no idea what will happen if test set has those outliers.
", "votes": 2, "replies": [ { "id": 2828477, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-22T06:19:39.457000", "content": "Before posting this, I debated myself whether it was better to keep this fact to myself and find a clever way to dodge the issue, or to point it out to the organizers and hosts and ask for fair treatment. In the end, I realized that the easiest path was to confront everyone with this fact and share the suffering.
", "votes": 10, "replies": [ { "id": 2828535, "author_name": "Federico Peccia", "author_url": "", "post_date": "2024-05-22T06:47:56.717000", "content": "So basically… the person/team that builds the model that is more robust to outliers will win?
", "votes": 0, "replies": [] }, { "id": 2828557, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-05-22T07:00:22.040000", "content": "Thanks for sharing. I think keeping information like this to yourself doesn't worth it. It may still backfire at the end since the metric is not stable. Changing metric to weighted macro average r2 score would be fine for this competition imo.
", "votes": 0, "replies": [] }, { "id": 2828584, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-22T07:10:11.527000", "content": "\n\nChanging metric to weighted macro average r2 score would be fine for this competition imo.
\n
Yeah. I think R2 (normalizing with unbounded sigma) as it is too extreme for this competition. At least custom metrics with normalizing clipped weight (sigma is clipped with some tiny value) should have been better?
", "votes": 0, "replies": [] }, { "id": 2828590, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-22T07:12:04.380000", "content": "\n\nSo basically… the person/team that builds the model that is more robust to outliers will win?
\n
Possibly. But it depends on the test set. If host already removed these outliers, winners in public LB highly possibly wins.
", "votes": 2, "replies": [] }, { "id": 2828609, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-22T07:19:45.567000", "content": "My other thought was to ask host to publish their unet baseline build for this comp and ask clarification for it surely have decent score on private LB. This would be a benchmark of safeness.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2881583, "author_name": "Juan D C F", "author_url": "", "post_date": "2024-06-20T19:43:53.260000", "content": "Cool plot, what tool did you use to generate it? Meaning, did you just use Pandas, or something more sophisticated like Apache Beam? Also, what kind of hardware did you use? Some times I run out of memory trying to analyze the whole dataset.
", "votes": 0, "replies": [ { "id": 2882300, "author_name": "Bilzard", "author_url": "", "post_date": "2024-06-21T09:27:22.100000", "content": "I used polars on 128 GB RAM PC.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2828449": "Hi. Hope you and your fellow Sisyphus are doing well with carrying the rock.\n\nI'll show some evidence of this competition's extreme setup.\n\n## TL; DR\n\nsome target columns are significantly risky.\nIf one single outlier sample could cause significant drop on private LB (~-12 with single sample).\n\n## How these graphs are created\n\nI plotted the risk of all 368 target columns except trivial columns with σ=0.\nThe risk is calculated by\n\n$$\n\\text{risk} := \\max(y) / \\sigma(y)\n$$\n\nThe first and 2nd picture shows some columns has significantly distant outliers (up to ~2500 σ).\n\nThis could cause your model significantly drops in private LB.\nThe 3rd picture depicts this outlier case.\nSuppose if your model predict single outlier in private test set with ptend_q0002_27 = 0 (because 99.9999% has <= 0 values), whose actual value is 2523σ, you will get `current score - 12.24` in private LB.\n\nSo hope your model can handle these outliers well.\n(I also hope host's already removed these miserable outliers from test set).\n\nGood luck!\n\n\n\n\n\n", "2834423": "Hello, thank you for providing this detailed analysis of the dataset and sharing your concerns. If it helps, the private leaderboard score is only half a percent different from the public leaderboard score for the u-net benchmark submission.", "2832998": "## Request for Some Action\n\nI think generally machine learning models under current competition metrics (R2) is prone to some kind of extreme outliers (i.e. 2523σ).\nIf this kind of extreme outliers are in private test set, it possibly causes catastrophic shakeup in the end of the competition.\n\nHopefully, we believe some safeguard may be helpful for all of us. like, \n\n1. clipping extreme error in prediction error like @martynoveduard proposed[1]\n2. Fixing to use customized R2 score with softer normalization policy like @shlomoron proposed[2]\n3. Providing baseline model's prediction (i.e. submission.csv) on test set and claim that it has decent score on private test set like I proposed[3]\n\nc.f.) I also posted mathematical explanation for instability in R2 metrics[4].\n\n@jerrylin96 @ashleychow \n\nHow hosts and Kaggle team think of this risk?\n\nThanks in advance.\n\n## Reference\n\n- [1] https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2829249\n- [2] https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2829800\n- [3] https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2829931\n- [4] https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2829926", "2829249": "amazing study\n\nsuggestion:\nmacro-avg(r2-score) -> macro-avg(clip(r2-score,0,1))\n\nthis way even if the participant doesn't handle the outlier, he is losing ~0.003 per-column compared to the perfectly predicted column (it hurts, but better than to have -10 score in private)", "2829050": "Let's calculate **how extreme the resolution this competition requires to our model**.\n\nThis can be done with maximum value without normalizing with std deviation.\nThe below table is statistics of target features **without normalizing with std deviation**.\nSo, 2523σ in `ptend_q0002_27` corresponds with 6.7934e-9 [kg/kg] in actual humidity unit.\nHow this amount is difficult to predict?\n\nSuppose density of dry air is 1 [kg/m^3], this amount corresponds with 6.7934 μg of vapor within 1 m^3.\nI don't know how difficult this is (I appreciate kind person with sufficient domain knowledge suggest us).\n\n## Reference\n\n- [1] https://en.wikipedia.org/wiki/Density_of_air\n\n", "2828467": "I created my validation scheme based on this fact. Some of the targets are not stable between different folds. I do a similar simulation on 625k sized splits and found that some of the folds can get negative r2 score on q0001 12 13 14, q0002 27 28, q0003 12 13 14 targets. I have no idea what will happen if test set has those outliers.", "2881583": "Cool plot, what tool did you use to generate it? Meaning, did you just use Pandas, or something more sophisticated like Apache Beam? Also, what kind of hardware did you use? Some times I run out of memory trying to analyze the whole dataset." } }