The overview of the competition has a CRUCIAL instruction hidden in the evaluation section:
\n\n\nWe will be using a custom R-squared metric for evaluation, but on a weighted solution. Prior to submitting your prediction, please multiply your prediction data element-wise by the data found in
\nsample_submission.csv, which serves dual purpose as both a \"sample submission\" and a \"weighting file\". This will weight all of the target variables by pressure and area and \"convert\" them to common units (watts per meter squared).
⚠️ This means that unless you multiply the predictions by the sample submission, the predictions will be way off from the targets in the test set.
\nDon't forget to do this, it can mean the difference between wasting a submission or ignoring a potential good idea!
", "messages": [ { "id": 2761272, "postDate": "2024-04-19T18:45:56.773Z", "content": "The overview of the competition has a CRUCIAL instruction hidden in the evaluation section:
\n\n\nWe will be using a custom R-squared metric for evaluation, but on a weighted solution. Prior to submitting your prediction, please multiply your prediction data element-wise by the data found in
\nsample_submission.csv, which serves dual purpose as both a \"sample submission\" and a \"weighting file\". This will weight all of the target variables by pressure and area and \"convert\" them to common units (watts per meter squared).
⚠️ This means that unless you multiply the predictions by the sample submission, the predictions will be way off from the targets in the test set.
\nDon't forget to do this, it can mean the difference between wasting a submission or ignoring a potential good idea!
", "rawMarkdown": "The overview of the competition has a CRUCIAL instruction hidden in the evaluation section:\n\n>We will be using a custom R-squared metric for evaluation, but on a weighted solution. **Prior to submitting your prediction, please multiply your prediction data element-wise by the data found in `sample_submission.csv`**, which serves dual purpose as both a \"sample submission\" and a \"weighting file\". This will weight all of the target variables by pressure and area and \"convert\" them to common units (watts per meter squared).\n\n⚠️ This means that *unless you multiply the predictions by the sample submission, the predictions will be way off from the targets in the test set*. \n\nDon't forget to do this, it can mean the difference between wasting a submission or ignoring a potential good idea!\n", "votes": 13 }, { "id": 2762989, "postDate": "2024-04-20T07:41:37.463Z", "content": "FYI This is for submission only. When I multiply weights with my predictions, my validation scores becomes worse.
", "rawMarkdown": "FYI This is for submission only. When I multiply weights with my predictions, my validation scores becomes worse.", "votes": 1, "replies": [ { "id": 2763024, "postDate": "2024-04-20T08:12:13.603Z", "content": "Yes, exactly. The targets in the train set are on different scales to those on the hidden test, so when applying validation techniques we're still predicting targets from the same unmodified train distribution. Good idea to make that clear :)
", "rawMarkdown": "Yes, exactly. The targets in the train set are on different scales to those on the hidden test, so when applying validation techniques we're still predicting targets from the same unmodified train distribution. Good idea to make that clear :)", "votes": 1 } ] }, { "id": 2761329, "postDate": "2024-04-19T19:19:19.383Z", "content": "I have never seen a competition like this in my time with Kaggle. I will remember this for a long time for sure 😀 @asarvazyan
", "rawMarkdown": "I have never seen a competition like this in my time with Kaggle. I will remember this for a long time for sure 😀 @asarvazyan ", "votes": 1, "replies": [ { "id": 2761338, "postDate": "2024-04-19T19:22:40.353Z", "content": "Agreed. Quite an… interesting choice.
\nI'm having trouble understanding why the organizers didn't just apply this multiplication as part of the evaluation scripts.
", "rawMarkdown": "Agreed. Quite an... interesting choice.\n\nI'm having trouble understanding why the organizers didn't just apply this multiplication as part of the evaluation scripts. ", "votes": 5, "replies": [ { "id": 2761376, "postDate": "2024-04-19T19:49:20.113Z", "content": "In cricket, we have a term named well-left. I think it is good to do this with this one too @asarvazyan
", "rawMarkdown": "In cricket, we have a term named **well-left**. I think it is good to do this with this one too @asarvazyan " }, { "id": 2875176, "postDate": "2024-06-16T23:42:21.110Z", "content": "Looking at the weights is pretty informative - so glad we got to see them, but sure is confusing me to see many of the shared notebooks where it's not real clear that they are doing a multiply.
", "rawMarkdown": "Looking at the weights is pretty informative - so glad we got to see them, but sure is confusing me to see many of the shared notebooks where it's not real clear that they are doing a multiply." } ] } ] }, { "id": 2785712, "postDate": "2024-05-01T02:15:28.287Z", "content": "I did not fully understand it.😭
\nI can train a model and run input x through the model to get output y.
\nDoes this mean that I should not directly make y into a submitted file?
", "rawMarkdown": "I did not fully understand it.😭\n\nI can train a model and run input x through the model to get output **y**.\n\nDoes this mean that I should not directly make **y** into a submitted file?", "replies": [ { "id": 2786014, "postDate": "2024-05-01T05:45:11.847Z", "content": "It's strange, isn't it? But yes you're right, you cannot submit your 'y' predictions directly. You have to multiply them by the numbers in sample_submission.csv, which are not representative samples at all, but weighting factors. As I understand it.
", "rawMarkdown": "It's strange, isn't it? But yes you're right, you cannot submit your 'y' predictions directly. You have to multiply them by the numbers in sample_submission.csv, which are not representative samples at all, but weighting factors. As I understand it." }, { "id": 2786122, "postDate": "2024-05-01T06:51:17.717Z", "content": "That's right. Usually, we just submit our predictions. Also, the sample submission file is usually a somewhat realistic submission file (although it could be something like all ones).
\nIn this competition, they wanted to give us a set of weights to multiply by our predictions. I guess someone decided since the weights would have the same number of columns as the sample submission, they'd just create one file and use it for both purposes.
", "rawMarkdown": "That's right. Usually, we just submit our predictions. Also, the sample submission file is usually a somewhat realistic submission file (although it could be something like all ones).\n\nIn this competition, they wanted to give us a set of weights to multiply by our predictions. I guess someone decided since the weights would have the same number of columns as the sample submission, they'd just create one file and use it for both purposes.", "votes": 1 } ] }, { "id": 2781248, "postDate": "2024-04-28T17:08:14.340Z", "content": "I really don't understand this weighting business. How can everything end up in W/m2 when some quantities have nothing to do with energy flow, e.g. rain rate (m/s) or heating tendency (K/s)? I think we need a more coherent explanation from the competition organisers. It would make far more sense to have us submit data that have the same units and scaling as the target columns in the training data, and only then have them weight our accuracy on those columns according to how much variance or importance they have. Am I just being stupid? 😅
\n\"Other values are scaled by the inverse of the standard deviation.\" -- meaning in the training data (which gives rise to the 'sample submission' values) or calculated from our own predictions (which wouldn't seem to make any sense at all)?
\nPS I did look at the referenced script. It just shows how the example submission file contains weighting values which are identical for all rows in any given column, the actual predictions being trashed along the way. What kind of \"sample\" is that supposed to be for us to follow as an example?!
\nhttps://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py
FYI This is for submission only. When I multiply weights with my predictions, my validation scores becomes worse.
", "votes": 1, "replies": [ { "id": 2763024, "author_name": "asarvazyan", "author_url": "", "post_date": "2024-04-20T08:12:13.603000", "content": "Yes, exactly. The targets in the train set are on different scales to those on the hidden test, so when applying validation techniques we're still predicting targets from the same unmodified train distribution. Good idea to make that clear :)
", "votes": 1, "replies": [] } ] }, { "id": 2761329, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-04-19T19:19:19.383000", "content": "I have never seen a competition like this in my time with Kaggle. I will remember this for a long time for sure 😀 @asarvazyan
", "votes": 1, "replies": [ { "id": 2761338, "author_name": "asarvazyan", "author_url": "", "post_date": "2024-04-19T19:22:40.353000", "content": "Agreed. Quite an… interesting choice.
\nI'm having trouble understanding why the organizers didn't just apply this multiplication as part of the evaluation scripts.
", "votes": 5, "replies": [ { "id": 2761376, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-04-19T19:49:20.113000", "content": "In cricket, we have a term named well-left. I think it is good to do this with this one too @asarvazyan
", "votes": 0, "replies": [] }, { "id": 2875176, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2024-06-16T23:42:21.110000", "content": "Looking at the weights is pretty informative - so glad we got to see them, but sure is confusing me to see many of the shared notebooks where it's not real clear that they are doing a multiply.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2785712, "author_name": "thoth000", "author_url": "", "post_date": "2024-05-01T02:15:28.287000", "content": "I did not fully understand it.😭
\nI can train a model and run input x through the model to get output y.
\nDoes this mean that I should not directly make y into a submitted file?
", "votes": 0, "replies": [ { "id": 2786014, "author_name": "Charlie Wartnaby", "author_url": "", "post_date": "2024-05-01T05:45:11.847000", "content": "It's strange, isn't it? But yes you're right, you cannot submit your 'y' predictions directly. You have to multiply them by the numbers in sample_submission.csv, which are not representative samples at all, but weighting factors. As I understand it.
", "votes": 0, "replies": [] }, { "id": 2786122, "author_name": "Ted K", "author_url": "", "post_date": "2024-05-01T06:51:17.717000", "content": "That's right. Usually, we just submit our predictions. Also, the sample submission file is usually a somewhat realistic submission file (although it could be something like all ones).
\nIn this competition, they wanted to give us a set of weights to multiply by our predictions. I guess someone decided since the weights would have the same number of columns as the sample submission, they'd just create one file and use it for both purposes.
", "votes": 1, "replies": [] } ] }, { "id": 2781248, "author_name": "Charlie Wartnaby", "author_url": "", "post_date": "2024-04-28T17:08:14.340000", "content": "I really don't understand this weighting business. How can everything end up in W/m2 when some quantities have nothing to do with energy flow, e.g. rain rate (m/s) or heating tendency (K/s)? I think we need a more coherent explanation from the competition organisers. It would make far more sense to have us submit data that have the same units and scaling as the target columns in the training data, and only then have them weight our accuracy on those columns according to how much variance or importance they have. Am I just being stupid? 😅
\n\"Other values are scaled by the inverse of the standard deviation.\" -- meaning in the training data (which gives rise to the 'sample submission' values) or calculated from our own predictions (which wouldn't seem to make any sense at all)?
\nPS I did look at the referenced script. It just shows how the example submission file contains weighting values which are identical for all rows in any given column, the actual predictions being trashed along the way. What kind of \"sample\" is that supposed to be for us to follow as an example?!
\nhttps://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py