When calculating the R2 score on validation data, using denormalized model outputs yields a very high R2 score, close to 0.9. However, using the normalized values directly aligns closely with LB scores. Can anyone explain the reason behind this discrepancy?
", "messages": [ { "id": 2886293, "postDate": "2024-06-23T15:03:57.163Z", "content": "When calculating the R2 score on validation data, using denormalized model outputs yields a very high R2 score, close to 0.9. However, using the normalized values directly aligns closely with LB scores. Can anyone explain the reason behind this discrepancy?
", "rawMarkdown": "When calculating the R2 score on validation data, using denormalized model outputs yields a very high R2 score, close to 0.9. However, using the normalized values directly aligns closely with LB scores. Can anyone explain the reason behind this discrepancy? ", "votes": 3 }, { "id": 2886761, "postDate": "2024-06-23T19:10:27.437Z", "content": "When using denormalized data do you use original float64 data as y_true or converted to float32?
", "rawMarkdown": "When using denormalized data do you use original float64 data as y_true or converted to float32?", "replies": [ { "id": 2886940, "postDate": "2024-06-23T23:54:20.623Z", "content": "Thank you for your comment. I normalize the data in float64 and then convert it to float32 for training.
", "rawMarkdown": "Thank you for your comment. I normalize the data in float64 and then convert it to float32 for training.", "replies": [ { "id": 2887332, "postDate": "2024-06-24T06:28:28.067Z", "content": "Try calculating R2 on denormalized predicts and original labels
", "rawMarkdown": "Try calculating R2 on denormalized predicts and original labels" }, { "id": 2887529, "postDate": "2024-06-24T08:48:39.627Z", "content": "have you fixed the problem,i use pytorch nn for the new weights，the valid r2 is 0.9x
", "rawMarkdown": "have you fixed the problem,i use pytorch nn for the new weights，the valid r2 is 0.9x" }, { "id": 2896175, "postDate": "2024-06-29T16:36:50.233Z", "content": "Me too. I am using pytorch nn, also not convert data to float32, try calculating R2 on denormalized predicts and original labels, and normalized them for training. I got 0.9x for valid and got only 0.3 for actual submission
", "rawMarkdown": "Me too. I am using pytorch nn, also not convert data to float32, try calculating R2 on denormalized predicts and original labels, and normalized them for training. I got 0.9x for valid and got only 0.3 for actual submission" }, { "id": 2902209, "postDate": "2024-07-03T06:13:10.037Z", "content": "I am in the same situation as well. What should we do? Is the dataset leaking?
", "rawMarkdown": "I am in the same situation as well. What should we do? Is the dataset leaking?" }, { "id": 2902231, "postDate": "2024-07-03T06:28:23.473Z", "content": "You should calculate R2 with the original unprocessed data.
", "rawMarkdown": "You should calculate R2 with the original unprocessed data." }, { "id": 2902273, "postDate": "2024-07-03T06:57:13.607Z", "content": "I managed to resolve the issue, although the root cause remains unclear. I recommend calculating the R2 score using Scikit-learn and then clipping the values between 0 and 1.
", "rawMarkdown": "I managed to resolve the issue, although the root cause remains unclear. I recommend calculating the R2 score using Scikit-learn and then clipping the values between 0 and 1." } ] } ] } ], "comments": [ { "id": 2886761, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-06-23T19:10:27.437000", "content": "When using denormalized data do you use original float64 data as y_true or converted to float32?
", "votes": 0, "replies": [ { "id": 2886940, "author_name": "kinohito", "author_url": "", "post_date": "2024-06-23T23:54:20.623000", "content": "Thank you for your comment. I normalize the data in float64 and then convert it to float32 for training.
", "votes": 0, "replies": [ { "id": 2887332, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-06-24T06:28:28.067000", "content": "Try calculating R2 on denormalized predicts and original labels
", "votes": 0, "replies": [] }, { "id": 2887529, "author_name": "yanqiangmiffy", "author_url": "", "post_date": "2024-06-24T08:48:39.627000", "content": "have you fixed the problem,i use pytorch nn for the new weights，the valid r2 is 0.9x
", "votes": 0, "replies": [] }, { "id": 2896175, "author_name": "pluk io", "author_url": "", "post_date": "2024-06-29T16:36:50.233000", "content": "Me too. I am using pytorch nn, also not convert data to float32, try calculating R2 on denormalized predicts and original labels, and normalized them for training. I got 0.9x for valid and got only 0.3 for actual submission
", "votes": 0, "replies": [] }, { "id": 2902209, "author_name": "TOUGH MIYAZAWA", "author_url": "", "post_date": "2024-07-03T06:13:10.037000", "content": "I am in the same situation as well. What should we do? Is the dataset leaking?
", "votes": 0, "replies": [] }, { "id": 2902231, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-07-03T06:28:23.473000", "content": "You should calculate R2 with the original unprocessed data.
", "votes": 0, "replies": [] }, { "id": 2902273, "author_name": "kinohito", "author_url": "", "post_date": "2024-07-03T06:57:13.607000", "content": "I managed to resolve the issue, although the root cause remains unclear. I recommend calculating the R2 score using Scikit-learn and then clipping the values between 0 and 1.
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2886293": "When calculating the R2 score on validation data, using denormalized model outputs yields a very high R2 score, close to 0.9. However, using the normalized values directly aligns closely with LB scores. Can anyone explain the reason behind this discrepancy? ", "2886761": "When using denormalized data do you use original float64 data as y_true or converted to float32?" } }