I know not many people make this mistake, but I'm sharing it here because I was mistake.
\nWhen calculating R2 score, be careful about the order of y_true and y_pred. This metric is asymmetric, unlike RMSE.
\nFor example,
\nin sklearn: r2_score(y_true, y_pred)
\nin torch metrics: r2_score(y_pred, y_true)
I had repeated the experiment with the wrong order and suffered from the score not increasing as expected. However, after correcting this, CV and LB scores improved from 0.43 to 0.58
\nThe difference in scores is due to post-processing that fills in the predictions to 0 when R2 score is less than 0, as shared in some public notebooks. (e.g., this notebook)
\nThe calculated my R2 score was incorrect, leading to scores unintentionally falling below 0 in several columns. This error caused to produce lower scores.
I know not many people make this mistake, but I'm sharing it here because I was mistake.
\nWhen calculating R2 score, be careful about the order of y_true and y_pred. This metric is asymmetric, unlike RMSE.
\nFor example,
\nin sklearn: r2_score(y_true, y_pred)
\nin torch metrics: r2_score(y_pred, y_true)
I had repeated the experiment with the wrong order and suffered from the score not increasing as expected. However, after correcting this, CV and LB scores improved from 0.43 to 0.58
\nThe difference in scores is due to post-processing that fills in the predictions to 0 when R2 score is less than 0, as shared in some public notebooks. (e.g., this notebook)
\nThe calculated my R2 score was incorrect, leading to scores unintentionally falling below 0 in several columns. This error caused to produce lower scores.
Hahaha I also made this mistake several weeks ago and there was a lot of frustration 🤣
", "rawMarkdown": "Hahaha I also made this mistake several weeks ago and there was a lot of frustration 🤣" }, { "id": 2812953, "postDate": "2024-05-14T13:45:48.737Z", "content": "Hello, I checked for sklearn-based notebook (Giba's) and it seems that the R² score is calculated properly as (true, pred). Where exactly do yo think there's something wrong ?
", "rawMarkdown": "Hello, I checked for sklearn-based notebook (Giba's) and it seems that the R² score is calculated properly as (true, pred). Where exactly do yo think there's something wrong ?", "replies": [ { "id": 2812987, "postDate": "2024-05-14T14:09:22.960Z", "content": "Yes, Giba's public notebook is correct.
\nI mean that post-processing based on r2 score with a miscalculation of r2_score is not good.
", "rawMarkdown": "Yes, Giba's public notebook is correct.\n\n I mean that post-processing based on r2 score **with** a miscalculation of r2_score is not good.", "votes": 1, "replies": [ { "id": 2814369, "postDate": "2024-05-15T09:08:59.800Z", "content": "Okay thank you ! Will definitely be careful about this
", "rawMarkdown": "Okay thank you ! Will definitely be careful about this" } ] } ] }, { "id": 2822367, "postDate": "2024-05-18T14:53:06.477Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2812955, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-14T13:47:56.500000", "content": "Hahaha I also made this mistake several weeks ago and there was a lot of frustration 🤣
", "votes": 0, "replies": [] }, { "id": 2812953, "author_name": "Albanito", "author_url": "", "post_date": "2024-05-14T13:45:48.737000", "content": "Hello, I checked for sklearn-based notebook (Giba's) and it seems that the R² score is calculated properly as (true, pred). Where exactly do yo think there's something wrong ?
", "votes": 0, "replies": [ { "id": 2812987, "author_name": "kuto", "author_url": "", "post_date": "2024-05-14T14:09:22.960000", "content": "Yes, Giba's public notebook is correct.
\nI mean that post-processing based on r2 score with a miscalculation of r2_score is not good.
", "votes": 1, "replies": [ { "id": 2814369, "author_name": "Albanito", "author_url": "", "post_date": "2024-05-15T09:08:59.800000", "content": "Okay thank you ! Will definitely be careful about this
", "votes": 0, "replies": [] } ] } ] }, { "id": 2822367, "author_name": "", "author_url": "", "post_date": "2024-05-18T14:53:06.477000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2812565": "I know not many people make this mistake, but I'm sharing it here because I was mistake.\n\nWhen calculating R2 score, be careful about the order of y_true and y_pred. This metric is asymmetric, unlike RMSE.\nFor example, \nin [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score): `r2_score(y_true, y_pred)`\nin [torch metrics](https://lightning.ai/docs/torchmetrics/stable/regression/r2_score.html#functional-interface): `r2_score(y_pred, y_true)`\n\nI had repeated the experiment with the wrong order and suffered from the score not increasing as expected. However, after correcting this, CV and LB scores improved from 0.43 to 0.58\n\nThe difference in scores is due to post-processing that fills in the predictions to 0 when R2 score is less than 0, as shared in some public notebooks. (e.g., [this notebook](https://www.kaggle.com/code/titericz/giba-baseline-xgboost))\nThe calculated my R2 score was incorrect, leading to scores unintentionally falling below 0 in several columns. This error caused to produce lower scores.", "2812955": "Hahaha I also made this mistake several weeks ago and there was a lot of frustration 🤣", "2812953": "Hello, I checked for sklearn-based notebook (Giba's) and it seems that the R² score is calculated properly as (true, pred). Where exactly do yo think there's something wrong ?", "2822367": "" } }