![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2F6c0d94c9ea3aac387b134492e6e95b4a%2FClipboard01.jpg?generation=1713546432746325&alt=media\")
I tried recreating the training data from the low-resolution data described in the Data explanation page.
\nfrom datasets import load_dataset\n\n load_dataset()\nBut the data download from Hugging Face doesn't seem to be finishing any time soon.
\nTherefore, when I checked GitHub, there is a description here that low-resolution real geography is as follows.
\n
From this description, the low-resolution data is 744GB. So, I understood this explanation in data explanation page.
\nGiven size data, we recommend downloading converting them more manageable (like parquet npy). If you have access High-Performance Computing (HPC) cluster sufficient storage availability, you can also download data directly your remote cluster Kaggle API following :\n\nkaggle competitions download -c ClimSim\nMaybe we should download each file, for example from low-resolution hugging face site and remake train data with this script.
\nOr, using kaggle dataset ( 192.3 Gb ).
\nAnyway, this competition seems to be testing the ability to process a lot of data.
\nPlease let me know if I have misunderstood anything.
Enjoy !!
", "messages": [ { "id": 2761154, "postDate": "2024-04-19T17:18:08.297Z", "content": "I tried recreating the training data from the low-resolution data described in the Data explanation page.
\nfrom datasets import load_dataset\n\n load_dataset()\nBut the data download from Hugging Face doesn't seem to be finishing any time soon.
\nTherefore, when I checked GitHub, there is a description here that low-resolution real geography is as follows.
\n
From this description, the low-resolution data is 744GB. So, I understood this explanation in data explanation page.
\nGiven size data, we recommend downloading converting them more manageable (like parquet npy). If you have access High-Performance Computing (HPC) cluster sufficient storage availability, you can also download data directly your remote cluster Kaggle API following :\n\nkaggle competitions download -c ClimSim\nMaybe we should download each file, for example from low-resolution hugging face site and remake train data with this script.
\nOr, using kaggle dataset ( 192.3 Gb ).
\nAnyway, this competition seems to be testing the ability to process a lot of data.
\nPlease let me know if I have misunderstood anything.
Enjoy !!
", "rawMarkdown": "I tried recreating the training data from the low-resolution data described in the Data explanation [page](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/data).\n~~~\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"LEAP/ClimSim_low-res\")\n~~~\n\nBut the data download from Hugging Face doesn't seem to be finishing any time soon.\n\nTherefore, when I checked [GitHub](https://leap-stc.github.io/ClimSim/dataset.html), there is a description here that low-resolution real geography is as follows.\n\n\n\nFrom this description, the low-resolution data is 744GB. So, I understood this explanation in data explanation [page](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/data).\n\n~~~\nGiven the size of the data, we recommend downloading one file at a time and converting them to a more manageable format (like parquet or npy). If you have access to a High-Performance Computing (HPC) cluster with sufficient storage availability, you can also download the data directly to your remote cluster using the Kaggle API with the following command:\n\nkaggle competitions download -c ClimSim\n~~~\n\nMaybe we should download each file, for example from [low-resolution hugging face site](https://huggingface.co/datasets/LEAP/ClimSim_low-res/tree/main/train) and remake train data with [this script](https://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py).\n\nOr, using kaggle dataset ( 192.3 Gb ).\n\nAnyway, this competition seems to be testing the ability to process a lot of data.\nPlease let me know if I have misunderstood anything.\n\nEnjoy !!", "votes": 13 }, { "id": 2761161, "postDate": "2024-04-19T17:20:50.917Z", "content": "@chumajin you are absolutely right. This data amounts to 740 Gb! I tried to work on this earlier today and was warded off due to the size of the data!
\nI am unsure how one may process this data at this size with the resources available at an individual's disposal. Perhaps a cap on the size of data considering the nature of the participant base should be considered.
@ravi20076 Thank you for comment! I also ended up not being able to download it. Yes, using all the data might be difficult on a typical machine. It seems that the key will be which data to use and how much of it to use.
", "rawMarkdown": "@ravi20076 Thank you for comment! I also ended up not being able to download it. Yes, using all the data might be difficult on a typical machine. It seems that the key will be which data to use and how much of it to use.", "votes": 3 }, { "id": 2766228, "postDate": "2024-04-21T15:01:08.917Z", "content": "Hey @ravi20076 i am doing batch processing for neural nets although i agree with some of your points i have tried some iterations with spark XGB too and it took more that 8 hours
\nbut in my opinion working with npy files might do the trick
Please rate my solution if helpful
\nhttps://www.kaggle.com/code/starcs2001/neural-nets-starter-with-batch-processing
@chumajin you are absolutely right. This data amounts to 740 Gb! I tried to work on this earlier today and was warded off due to the size of the data!
\nI am unsure how one may process this data at this size with the resources available at an individual's disposal. Perhaps a cap on the size of data considering the nature of the participant base should be considered.
@ravi20076 Thank you for comment! I also ended up not being able to download it. Yes, using all the data might be difficult on a typical machine. It seems that the key will be which data to use and how much of it to use.
", "votes": 3, "replies": [] }, { "id": 2766228, "author_name": "Satej Raste", "author_url": "", "post_date": "2024-04-21T15:01:08.917000", "content": "Hey @ravi20076 i am doing batch processing for neural nets although i agree with some of your points i have tried some iterations with spark XGB too and it took more that 8 hours
\nbut in my opinion working with npy files might do the trick
Please rate my solution if helpful
\nhttps://www.kaggle.com/code/starcs2001/neural-nets-starter-with-batch-processing