![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2Fdcdfef4534835d2f46668af6c290c2f5%2FClipboard03.jpg?generation=1713552587674532&alt=media\")
I will share some code that seems to be useful for this competition.
\nThis competition involves large data sets, and just opening or saving them can be time-consuming. Therefore, using polars can help reduce time and make the process more comfortable.
polars pl\nsample = pl.read_csv(\"/kaggle/input/leap-atmospheric-physics-ai-climsim/sample_submission.csv\",n_rows=)\n# \ntargets = sample.[:]\n## change pandas polars you use pandas\ntest_polars = pl.from\ntest_polars.write\n
I will share some code that seems to be useful for this competition.
\nThis competition involves large data sets, and just opening or saving them can be time-consuming. Therefore, using polars can help reduce time and make the process more comfortable.
polars pl\nsample = pl.read_csv(\"/kaggle/input/leap-atmospheric-physics-ai-climsim/sample_submission.csv\",n_rows=)\n# \ntargets = sample.[:]\n## change pandas polars you use pandas\ntest_polars = pl.from\ntest_polars.write\n
You can also convert .csv to .parquet with low RAM
\nJust use lazy execution of polars, -
import polars pl\n\nstorage_dir = \n file os.listdir(storage_dir):\n df = pl.scan\n df.sink)\n@martynoveduard Thank you. Indeed, converting to Parquet when saving small files seems useful. Thank you for including lazy processing as well. It will be helpful.
", "rawMarkdown": "@martynoveduard Thank you. Indeed, converting to Parquet when saving small files seems useful. Thank you for including lazy processing as well. It will be helpful.", "replies": [ { "id": 2768917, "postDate": "2024-04-23T04:53:36.313Z", "content": "In the end I went with this code:
\n tqdm\n os\n\nos.makedirs(storage_dir / )\nframe_size = _000 # total length _091_520\n chunk_index tqdm.tqdm(range(, (num_train_samples // frame_size) + )):\n = frame_size * chunk_index\n df_sub = df.(, frame_size)\n df_sub.sink_parquet(storage_dir / / f)\nAfter, during dataset creation, you can also lazily load like this:
\n [pl.scan_parquet(data_dir / / f) for i in indices]\n pl.concat(dfs) \nWorks fast!
", "rawMarkdown": "In the end I went with this code:\n```\nimport tqdm\nimport os\n\nos.makedirs(storage_dir / 'train_chunks')\nframe_size = 200_000 # total length is 10_091_520\nfor chunk_index in tqdm.tqdm(range(0, (num_train_samples // frame_size) + 1)):\n offset = frame_size * chunk_index\n df_sub = df.slice(offset, frame_size)\n df_sub.sink_parquet(storage_dir / 'train_chunks' / f'train_{chunk_index}.parquet')\n```\n\nAfter, during dataset creation, you can also lazily load like this:\n```\ndfs = [pl.scan_parquet(data_dir / \"train_chunks\" / f\"train_{i}.parquet\") for i in indices]\ndf = pl.concat(dfs) \n```\nWorks fast!", "votes": 10 }, { "id": 2769189, "postDate": "2024-04-23T07:52:32.113Z", "content": "@martynoveduard Feels good! Thank you!!
", "rawMarkdown": "@martynoveduard Feels good! Thank you!!" } ] } ] }, { "id": 2761330, "postDate": "2024-04-19T19:20:16.887Z", "content": "Try DuckDB also, I think it is also a good option for such large datasets @chumajin
", "rawMarkdown": "Try DuckDB also, I think it is also a good option for such large datasets @chumajin ", "votes": 1, "replies": [ { "id": 2768633, "postDate": "2024-04-23T00:52:14.440Z", "content": "@ravi20076 Thank you for the addition. Yes, I have seen DuckDB performing well in other competitions as well. It will be helpful in this competition too.
", "rawMarkdown": "@ravi20076 Thank you for the addition. Yes, I have seen DuckDB performing well in other competitions as well. It will be helpful in this competition too." } ] }, { "id": 2761352, "postDate": "2024-04-19T19:32:55.503Z", "content": "Hi chumajin,
\nOne of the things that contributes to the speed of the Polars library is that it is in operation Eager and Lazy Execution options and It is derived from the Rust programming language.
\nPolars that follow the Lazy Execution system Unlike
\nPandas that follow the Eager Execution system
\n@chumajin
@adnanalaref Hi. You are right. Thank you for comment !
", "rawMarkdown": "@adnanalaref Hi. You are right. Thank you for comment !" } ] } ], "comments": [ { "id": 2762755, "author_name": "slime", "author_url": "", "post_date": "2024-04-20T04:36:21.537000", "content": "You can also convert .csv to .parquet with low RAM
\nJust use lazy execution of polars, -
import polars pl\n\nstorage_dir = \n file os.listdir(storage_dir):\n df = pl.scan\n df.sink)\n@martynoveduard Thank you. Indeed, converting to Parquet when saving small files seems useful. Thank you for including lazy processing as well. It will be helpful.
", "votes": 0, "replies": [ { "id": 2768917, "author_name": "slime", "author_url": "", "post_date": "2024-04-23T04:53:36.313000", "content": "In the end I went with this code:
\n tqdm\n os\n\nos.makedirs(storage_dir / )\nframe_size = _000 # total length _091_520\n chunk_index tqdm.tqdm(range(, (num_train_samples // frame_size) + )):\n = frame_size * chunk_index\n df_sub = df.(, frame_size)\n df_sub.sink_parquet(storage_dir / / f)\nAfter, during dataset creation, you can also lazily load like this:
\n [pl.scan_parquet(data_dir / / f) for i in indices]\n pl.concat(dfs) \nWorks fast!
", "votes": 10, "replies": [] }, { "id": 2769189, "author_name": "chumajin", "author_url": "", "post_date": "2024-04-23T07:52:32.113000", "content": "@martynoveduard Feels good! Thank you!!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2761330, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-04-19T19:20:16.887000", "content": "Try DuckDB also, I think it is also a good option for such large datasets @chumajin
", "votes": 1, "replies": [ { "id": 2768633, "author_name": "chumajin", "author_url": "", "post_date": "2024-04-23T00:52:14.440000", "content": "@ravi20076 Thank you for the addition. Yes, I have seen DuckDB performing well in other competitions as well. It will be helpful in this competition too.
", "votes": 0, "replies": [] } ] }, { "id": 2761352, "author_name": "Adnan Alaref", "author_url": "", "post_date": "2024-04-19T19:32:55.503000", "content": "Hi chumajin,
\nOne of the things that contributes to the speed of the Polars library is that it is in operation Eager and Lazy Execution options and It is derived from the Rust programming language.
\nPolars that follow the Lazy Execution system Unlike
\nPandas that follow the Eager Execution system
\n@chumajin
@adnanalaref Hi. You are right. Thank you for comment !
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2761282": "I will share some code that seems to be useful for this competition.\nThis competition involves large data sets, and just opening or saving them can be time-consuming. Therefore, using polars can help reduce time and make the process more comfortable.\n\n## Targets extraction\n\n~~~\nimport polars as pl\nsample = pl.read_csv(\"/kaggle/input/leap-atmospheric-physics-ai-climsim/sample_submission.csv\",n_rows=1)\n# read only 1 rows\ntargets = sample.columns[1:]\n~~~\n\n## Save submission.csv\n\n~~~\n## change pandas to polars if you use pandas\ntest_polars = pl.from_pandas(test[[\"sample_id\"]+targets])\ntest_polars.write_csv(\"submission.csv\")\n~~~\n\n## Result of time comparison between polars and pandas\n\n\n\n## **About 18 times faster using by polars.**", "2762755": "You can also convert .csv to .parquet with low RAM\nJust use lazy execution of polars, - \n```\nimport polars as pl\n\nstorage_dir = Path('/path/to/dataset/')\nfor file in os.listdir(storage_dir):\n df = pl.scan_csv(storage_dir / file)\n df.sink_parquet(storage_dir / file.replace('.csv', '.parquet'))\n```", "2761330": "Try DuckDB also, I think it is also a good option for such large datasets @chumajin ", "2761352": "Hi chumajin,\nOne of the things that contributes to the speed of the Polars library is that it is in operation `Eager and Lazy Execution options` and It is derived from the `Rust programming language`.\n`Polars that follow the Lazy Execution system` Unlike\n`Pandas that follow the Eager Execution system`\n@chumajin " } }