![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fbd45e1887efb329c5ce4a57323dcf73c%2FScreenshot%202024-04-21%20at%2010.45.07.png?generation=1713663950689154&alt=media\")
\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F09a47aac73a7c406946720e1296164f3%2FScreenshot%202024-04-21%20at%2010.57.59.png?generation=1713664719789206&alt=media\")
I want to ask host and Kaggle team about the task design.
\nI briefly look through data tab, and found the latitude, longitude, and timestamp data is dropped on both train and test data. And I wonder why.
\nObviously, feeding these information will allow us to design more various modeling (e.g. time series modeling, prediction using input feature of nearby location etc.).
\nI think adding these information won't contradict to the competition's objective to design cheap ML models to simulate climate dynamics.
\nAre there any possibility of adding them?
\n", "messages": [ { "id": 2763209, "postDate": "2024-04-20T11:41:17.120Z", "content": "I want to ask host and Kaggle team about the task design.
\nI briefly look through data tab, and found the latitude, longitude, and timestamp data is dropped on both train and test data. And I wonder why.
\nObviously, feeding these information will allow us to design more various modeling (e.g. time series modeling, prediction using input feature of nearby location etc.).
\nI think adding these information won't contradict to the competition's objective to design cheap ML models to simulate climate dynamics.
\nAre there any possibility of adding them?
\n", "rawMarkdown": "I want to ask host and Kaggle team about the task design.\n\nI briefly look through data tab, and found the latitude, longitude, and timestamp data is dropped on both train and test data. And I wonder why.\n\nObviously, feeding these information will allow us to design more various modeling (e.g. time series modeling, prediction using input feature of nearby location etc.).\n\nI think adding these information won't contradict to the competition's objective to design cheap ML models to simulate climate dynamics.\n\nAre there any possibility of adding them?\n\n@ashleychow, @mylesoneill", "votes": 30 }, { "id": 2768949, "postDate": "2024-04-23T05:13:57.213Z", "content": "Hi there. This is a great question, and apologies for not addressing it from the get-go. The idea is to design ML emulators capable of closely emulating Cloud-Resolving Models (CRMs) which are not \"given\" information regarding lat/lon/time. ML emulators that properly learn the physics can hopefully generalize to out-of-sample climates; however, we would ideally like to ensure that they are actually learning the physics rather than geographic patterns.
", "rawMarkdown": "Hi there. This is a great question, and apologies for not addressing it from the get-go. The idea is to design ML emulators capable of closely emulating Cloud-Resolving Models (CRMs) which are not \"given\" information regarding lat/lon/time. ML emulators that properly learn the physics can hopefully generalize to out-of-sample climates; however, we would ideally like to ensure that they are actually learning the physics rather than geographic patterns.", "votes": 12, "replies": [ { "id": 2769003, "postDate": "2024-04-23T05:44:26.057Z", "content": "\n\nwe would ideally like to ensure that they are actually learning the physics rather than geographic patterns.
\n
Totally make sense. Thanks.
", "rawMarkdown": "> we would ideally like to ensure that they are actually learning the physics rather than geographic patterns.\n\nTotally make sense. Thanks.", "votes": 1 }, { "id": 2788391, "postDate": "2024-05-02T08:07:52.043Z", "content": "\n\nML emulators that properly learn the physics can hopefully generalize to out-of-sample climates
\n
I think training data consist of all seasons since it is too large but should we expect unseen locations in test set?
", "rawMarkdown": "> ML emulators that properly learn the physics can hopefully generalize to out-of-sample climates\n\nI think training data consist of all seasons since it is too large but should we expect unseen locations in test set?", "votes": 4 } ] }, { "id": 2800888, "postDate": "2024-05-08T11:51:44.057Z", "content": "While timestamp and location are not given as part of the official competition data, they can be trivially inferred from the original (https://huggingface.co/datasets/LEAP/ClimSim_low-res) data: there are 384 locations, all in order, with coordinated given in \"ClimSim_low-res_grid-info.nc\" file, and timestamp is a part of each filename, so is also available. And test data comes in the same format, so also follows the same pattern.
\nAnd location has a large impact on predictability: some locations produce R2 of > 95%, while others struggle to produce any positive R2.
\nI strongly suspect that all the top models use location and/or time information - i think that is the only way to generate material improvements in results vs \"unet_preds.csv\" benchmark.
", "rawMarkdown": "While timestamp and location are not given as part of the official competition data, they can be trivially inferred from the original (https://huggingface.co/datasets/LEAP/ClimSim_low-res) data: there are 384 locations, all in order, with coordinated given in \"ClimSim_low-res_grid-info.nc\" file, and timestamp is a part of each filename, so is also available. And test data comes in the same format, so also follows the same pattern.\n\nAnd location has a large impact on predictability: some locations produce R2 of > 95%, while others struggle to produce any positive R2.\n\nI strongly suspect that all the top models use location and/or time information - i think that is the only way to generate material improvements in results vs \"unet_preds.csv\" benchmark.", "votes": 3, "replies": [ { "id": 2800927, "postDate": "2024-05-08T12:16:53.617Z", "content": "I can confirm that score >0.70 can be achieved with only kaggle dataset (no external data used, no reverse engineering)
", "rawMarkdown": "I can confirm that score >0.70 can be achieved with only kaggle dataset (no external data used, no reverse engineering)", "votes": 8, "replies": [ { "id": 2801053, "postDate": "2024-05-08T13:06:48.440Z", "content": ">0.70 is nice, but how about your >0.75, also no rev. engineering? 😅
", "rawMarkdown": "\\>0.70 is nice, but how about your \\>0.75, also no rev. engineering? 😅" }, { "id": 2801064, "postDate": "2024-05-08T13:11:01.543Z", "content": "Yes! As for now, my score is kaggle dataset only
", "rawMarkdown": "Yes! As for now, my score is kaggle dataset only", "votes": 3 }, { "id": 2801073, "postDate": "2024-05-08T13:16:41.797Z", "content": "This is very helpful to know! And very good job on the LB, I anticipate you become GM after this comp. 💯
", "rawMarkdown": "This is very helpful to know! And very good job on the LB, I anticipate you become GM after this comp. 💯", "votes": 2 } ] } ] }, { "id": 2764074, "postDate": "2024-04-20T21:27:42.203Z", "content": "If I'm not mistaken, the data for each row is self-contained for the purpose of local physics calculations i.e. the physical calculations don't need the surrounding points in time and space for exact calculation of the target variables.
\nSo this is a good design of the competition imo.
I think this constraint (without space and time inputs) is optional because authors mentioned in their paper another task design with temporal locality.
\n
\n
You're correct there are a few artificial constraints as part of this competition introduced for the sake of simplicity. Spatially and temporally non-local information could indeed be beneficial, and it's possible that an operational emulator will make use of it. However, we also believe substantial headroom exists on the architecture front and it's still very possible that the innovations discovered from this Kaggle competition can make their way to something used operationally.
", "rawMarkdown": "You're correct there are a few artificial constraints as part of this competition introduced for the sake of simplicity. Spatially and temporally non-local information could indeed be beneficial, and it's possible that an operational emulator will make use of it. However, we also believe substantial headroom exists on the architecture front and it's still very possible that the innovations discovered from this Kaggle competition can make their way to something used operationally. ", "votes": 2 }, { "id": 2768993, "postDate": "2024-04-23T05:40:21.453Z", "content": "@jerrylin96 I got it. Thank you for clarification!
", "rawMarkdown": "@jerrylin96 I got it. Thank you for clarification!", "votes": 1 } ] } ] }, { "id": 2831363, "postDate": "2024-05-23T16:32:53.617Z", "content": "Just to confirm, are we allowed to develop a model that is aware of spatial/temporal information? (This is assuming that the grid info can be recovered from the test csv file).
", "rawMarkdown": "Just to confirm, are we allowed to develop a model that is aware of spatial/temporal information? (This is assuming that the grid info can be recovered from the test csv file).", "votes": 1, "replies": [ { "id": 2831372, "postDate": "2024-05-23T16:35:52.287Z", "content": "I don't see any reason it would not be allowed. BTW time stamp is easy to recover when looking at rolling mean of the features. Grid might be a bit harder.
", "rawMarkdown": "I don't see any reason it would not be allowed. BTW time stamp is easy to recover when looking at rolling mean of the features. Grid might be a bit harder." } ] }, { "id": 2765769, "postDate": "2024-04-21T10:39:35.103Z", "content": "I've also been thinking about designing a model that incorporates both spatial and temporal context :/
\nBut since we already have access to the final test set in this competition, it's possible that the coordinates and timestamps are excluded such that we cannot simply look up the target values using external climate data sources. Just a guess though…
\n\nit's possible that the coordinates and timestamps are excluded such that we cannot simply look up the target values using external climate data sources.
\n
I think we don't have to care about this leak, because according to the host's paper, the train/test data are generated with intense computations of simulation. Its possible to reproduce strictly saying, but maybe no-one want pay such amount of GPU computation for this competition.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fe72ac38895f031837de69edc031e2526%2FScreenshot%202024-04-21%20at%2022.08.31.png?generation=1713704931151402&alt=media\")
The test set uses held-out data not publicly available (on hugging face or otherwise).
", "rawMarkdown": "The test set uses held-out data not publicly available (on hugging face or otherwise). ", "votes": 5 } ] } ] }, { "id": 2765493, "postDate": "2024-04-21T06:58:10.583Z", "content": "This should be addressed, otherwise it would be hard to match the performance of E3SM-MMF model
", "rawMarkdown": "This should be addressed, otherwise it would be hard to match the performance of E3SM-MMF model", "votes": 1, "replies": [ { "id": 2768975, "postDate": "2024-04-23T05:27:50.767Z", "content": "Please see this comment:
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/495258#2768969
", "rawMarkdown": "Please see this comment:\n\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/495258#2768969", "votes": 1 } ] }, { "id": 2857724, "postDate": "2024-06-06T05:27:11.687Z", "content": "Does the lack of temporal information influence the validation way ?
\nIn host's paper, they split train / test by the temporal information, so it would be better to validate in the same way,
\nthough the datasets in the paper may be different from kaggle dataset.
The following sentence is in3 ClimSimDataset Construction of the paper.
\n", "rawMarkdown": "Does the lack of temporal information influence the validation way ?\nIn host's paper, they split train / test by the temporal information, so it would be better to validate in the same way,\nthough the datasets in the paper may be different from kaggle dataset.\n\nThe following sentence is in` 3 ClimSimDataset Construction` of the paper.\n>Dataset Split: The 10-year datasets are divided into: (a) a training and validation spanning the\n first 8 years (0001-02 to 0009-01; YYYY-MM), excluding the first simulated month for numerical\n spin-up, and (b) a test set spanning the remaining two years (i.e., 0009-03 to 0011-02). A one-month\n gap is intentionally introduced between the two sets to prevent test set contamination via temporal\n correlation. Both sets are stored separately in our data repositories." }, { "id": 2814740, "postDate": "2024-05-15T14:20:42.957Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2871206, "postDate": "2024-06-14T05:46:21.693Z", "content": "Dataset Split: The 10-year datasets are divided into: (a) a training and validation spanning the
\n
\n first 8 years (0001-02 to 0009-01; YYYY-MM), excluding the first simulated month for numerical
\n spin-up, and (b) a test set spanning the remaining two years (i.e., 0009-03 to 0011-02). A one-month
\n gap is intentionally introduced between the two sets to prevent test set contamination via temporal
\n correlation. Both sets are stored separately in our data repositories.
Thank you for support
", "rawMarkdown": "Thank you for support " } ], "comments": [ { "id": 2768949, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-04-23T05:13:57.213000", "content": "Hi there. This is a great question, and apologies for not addressing it from the get-go. The idea is to design ML emulators capable of closely emulating Cloud-Resolving Models (CRMs) which are not \"given\" information regarding lat/lon/time. ML emulators that properly learn the physics can hopefully generalize to out-of-sample climates; however, we would ideally like to ensure that they are actually learning the physics rather than geographic patterns.
", "votes": 12, "replies": [ { "id": 2769003, "author_name": "Bilzard", "author_url": "", "post_date": "2024-04-23T05:44:26.057000", "content": "\n\nwe would ideally like to ensure that they are actually learning the physics rather than geographic patterns.
\n
Totally make sense. Thanks.
", "votes": 1, "replies": [] }, { "id": 2788391, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-05-02T08:07:52.043000", "content": "\n\nML emulators that properly learn the physics can hopefully generalize to out-of-sample climates
\n
I think training data consist of all seasons since it is too large but should we expect unseen locations in test set?
", "votes": 4, "replies": [] } ] }, { "id": 2800888, "author_name": "Youri Matiounine", "author_url": "", "post_date": "2024-05-08T11:51:44.057000", "content": "While timestamp and location are not given as part of the official competition data, they can be trivially inferred from the original (https://huggingface.co/datasets/LEAP/ClimSim_low-res) data: there are 384 locations, all in order, with coordinated given in \"ClimSim_low-res_grid-info.nc\" file, and timestamp is a part of each filename, so is also available. And test data comes in the same format, so also follows the same pattern.
\nAnd location has a large impact on predictability: some locations produce R2 of > 95%, while others struggle to produce any positive R2.
\nI strongly suspect that all the top models use location and/or time information - i think that is the only way to generate material improvements in results vs \"unet_preds.csv\" benchmark.
", "votes": 3, "replies": [ { "id": 2800927, "author_name": "slime", "author_url": "", "post_date": "2024-05-08T12:16:53.617000", "content": "I can confirm that score >0.70 can be achieved with only kaggle dataset (no external data used, no reverse engineering)
", "votes": 8, "replies": [ { "id": 2801053, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-08T13:06:48.440000", "content": ">0.70 is nice, but how about your >0.75, also no rev. engineering? 😅
", "votes": 0, "replies": [] }, { "id": 2801064, "author_name": "slime", "author_url": "", "post_date": "2024-05-08T13:11:01.543000", "content": "Yes! As for now, my score is kaggle dataset only
", "votes": 3, "replies": [] }, { "id": 2801073, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-08T13:16:41.797000", "content": "This is very helpful to know! And very good job on the LB, I anticipate you become GM after this comp. 💯
", "votes": 2, "replies": [] } ] } ] }, { "id": 2764074, "author_name": "greySnow", "author_url": "", "post_date": "2024-04-20T21:27:42.203000", "content": "If I'm not mistaken, the data for each row is self-contained for the purpose of local physics calculations i.e. the physical calculations don't need the surrounding points in time and space for exact calculation of the target variables.
\nSo this is a good design of the competition imo.
I think this constraint (without space and time inputs) is optional because authors mentioned in their paper another task design with temporal locality.
\n
\n
You're correct there are a few artificial constraints as part of this competition introduced for the sake of simplicity. Spatially and temporally non-local information could indeed be beneficial, and it's possible that an operational emulator will make use of it. However, we also believe substantial headroom exists on the architecture front and it's still very possible that the innovations discovered from this Kaggle competition can make their way to something used operationally.
", "votes": 2, "replies": [] }, { "id": 2768993, "author_name": "Bilzard", "author_url": "", "post_date": "2024-04-23T05:40:21.453000", "content": "@jerrylin96 I got it. Thank you for clarification!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2831363, "author_name": "Renu Singh", "author_url": "", "post_date": "2024-05-23T16:32:53.617000", "content": "Just to confirm, are we allowed to develop a model that is aware of spatial/temporal information? (This is assuming that the grid info can be recovered from the test csv file).
", "votes": 1, "replies": [ { "id": 2831372, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-23T16:35:52.287000", "content": "I don't see any reason it would not be allowed. BTW time stamp is easy to recover when looking at rolling mean of the features. Grid might be a bit harder.
", "votes": 0, "replies": [] } ] }, { "id": 2765769, "author_name": "Felix Yang", "author_url": "", "post_date": "2024-04-21T10:39:35.103000", "content": "I've also been thinking about designing a model that incorporates both spatial and temporal context :/
\nBut since we already have access to the final test set in this competition, it's possible that the coordinates and timestamps are excluded such that we cannot simply look up the target values using external climate data sources. Just a guess though…
\n\nit's possible that the coordinates and timestamps are excluded such that we cannot simply look up the target values using external climate data sources.
\n
I think we don't have to care about this leak, because according to the host's paper, the train/test data are generated with intense computations of simulation. Its possible to reproduce strictly saying, but maybe no-one want pay such amount of GPU computation for this competition.
\n
The test set uses held-out data not publicly available (on hugging face or otherwise).
", "votes": 5, "replies": [] } ] } ] }, { "id": 2765493, "author_name": "slime", "author_url": "", "post_date": "2024-04-21T06:58:10.583000", "content": "This should be addressed, otherwise it would be hard to match the performance of E3SM-MMF model
", "votes": 1, "replies": [ { "id": 2768975, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-04-23T05:27:50.767000", "content": "Please see this comment:
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/495258#2768969
", "votes": 1, "replies": [] } ] }, { "id": 2857724, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-06-06T05:27:11.687000", "content": "Does the lack of temporal information influence the validation way ?
\nIn host's paper, they split train / test by the temporal information, so it would be better to validate in the same way,
\nthough the datasets in the paper may be different from kaggle dataset.
The following sentence is in3 ClimSimDataset Construction of the paper.
\n", "votes": 0, "replies": [] }, { "id": 2814740, "author_name": "", "author_url": "", "post_date": "2024-05-15T14:20:42.957000", "content": "", "votes": 0, "replies": [] }, { "id": 2871206, "author_name": "Harshita Sapra", "author_url": "", "post_date": "2024-06-14T05:46:21.693000", "content": "Dataset Split: The 10-year datasets are divided into: (a) a training and validation spanning the
\n
\n first 8 years (0001-02 to 0009-01; YYYY-MM), excluding the first simulated month for numerical
\n spin-up, and (b) a test set spanning the remaining two years (i.e., 0009-03 to 0011-02). A one-month
\n gap is intentionally introduced between the two sets to prevent test set contamination via temporal
\n correlation. Both sets are stored separately in our data repositories.
Thank you for support
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2763209": "I want to ask host and Kaggle team about the task design.\n\nI briefly look through data tab, and found the latitude, longitude, and timestamp data is dropped on both train and test data. And I wonder why.\n\nObviously, feeding these information will allow us to design more various modeling (e.g. time series modeling, prediction using input feature of nearby location etc.).\n\nI think adding these information won't contradict to the competition's objective to design cheap ML models to simulate climate dynamics.\n\nAre there any possibility of adding them?\n\n@ashleychow, @mylesoneill", "2768949": "Hi there. This is a great question, and apologies for not addressing it from the get-go. The idea is to design ML emulators capable of closely emulating Cloud-Resolving Models (CRMs) which are not \"given\" information regarding lat/lon/time. ML emulators that properly learn the physics can hopefully generalize to out-of-sample climates; however, we would ideally like to ensure that they are actually learning the physics rather than geographic patterns.", "2800888": "While timestamp and location are not given as part of the official competition data, they can be trivially inferred from the original (https://huggingface.co/datasets/LEAP/ClimSim_low-res) data: there are 384 locations, all in order, with coordinated given in \"ClimSim_low-res_grid-info.nc\" file, and timestamp is a part of each filename, so is also available. And test data comes in the same format, so also follows the same pattern.\n\nAnd location has a large impact on predictability: some locations produce R2 of > 95%, while others struggle to produce any positive R2.\n\nI strongly suspect that all the top models use location and/or time information - i think that is the only way to generate material improvements in results vs \"unet_preds.csv\" benchmark.", "2764074": "If I'm not mistaken, the data for each row is self-contained for the purpose of local physics calculations i.e. the physical calculations don't need the surrounding points in time and space for exact calculation of the target variables.\nSo this is a good design of the competition imo.", "2831363": "Just to confirm, are we allowed to develop a model that is aware of spatial/temporal information? (This is assuming that the grid info can be recovered from the test csv file).", "2765769": "I've also been thinking about designing a model that incorporates both spatial and temporal context :/\nBut since we already have access to the final test set in this competition, it's possible that the coordinates and timestamps are excluded such that we cannot simply look up the target values using external climate data sources. Just a guess though...", "2765493": "This should be addressed, otherwise it would be hard to match the performance of E3SM-MMF model", "2857724": "Does the lack of temporal information influence the validation way ?\nIn host's paper, they split train / test by the temporal information, so it would be better to validate in the same way,\nthough the datasets in the paper may be different from kaggle dataset.\n\nThe following sentence is in` 3 ClimSimDataset Construction` of the paper.\n>Dataset Split: The 10-year datasets are divided into: (a) a training and validation spanning the\n first 8 years (0001-02 to 0009-01; YYYY-MM), excluding the first simulated month for numerical\n spin-up, and (b) a test set spanning the remaining two years (i.e., 0009-03 to 0011-02). A one-month\n gap is intentionally introduced between the two sets to prevent test set contamination via temporal\n correlation. Both sets are stored separately in our data repositories.", "2814740": "", "2871206": "Thank you for support " } }