{ "id": 494968, "title": "Starter references and onboarding materials ", "url": "/competitions/leap-atmospheric-physics-ai-climsim/discussion/494968", "author_name": "Ravi Ramakrishnan", "post_date": "2024-04-19T06:21:52.089000", "votes": 43, "comment_count": 14, "views": 0, "content": "

Hello everyone,

\n

Wishing you the best for the upcoming competition! I have handpicked a few resources hoping they will aid you in onboarding for this challenge-

\n

Domain based research papers and reading materials

\n
    \n
  1. https://paperswithcode.com/paper/deepphysinet-bridging-deep-learning-and
    2. \n
  2. https://paperswithcode.com/paper/climax-a-foundation-model-for-weather-and
    3. \n
  3. https://physics.paperswithcode.com/paper/purely-data-driven-medium-range-weather
    4. \n
  4. https://physics.paperswithcode.com/paper/open-source-qgis-toolkit-for-the-advanced
    5. \n
  5. https://paperswithcode.com/paper/spatio-temporal-weather-forecasting-and
    6. \n
  6. https://paperswithcode.com/paper/ace-a-fast-skillful-learned-global
    7. \n
  7. https://paperswithcode.com/paper/climsim-a-large-multi-scale-dataset-for
    8. \n
  8. https://ieeexplore.ieee.org/abstract/document/9139804/
    9. \n
  9. https://ieeexplore.ieee.org/abstract/document/9996169
    10. \n
  10. https://proceedings.neurips.cc/paper_files/paper/2023/hash/45fbcc01349292f5e059a0b8b02c8c3f-Abstract-Datasets_and_Benchmarks.html
    11. \n
  11. https://arxiv.org/abs/2306.08754
    12. \n
\n

This assignment is a tabular dataset with a very large size. 192 GB is a humungous dataset size for one to manipulate. As a word of caution, be mindful of the amount of computational resources available at your end while making a suitable decision to participate in the challenge. Also, kindly note that one may be able to extend the training dataset using the link below. Please peruse the overview page for more details-

\n\n

Manipulating large datasets

\n

DuckDB

\n
    \n
  1. https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest
    2. \n
  2. https://www.kaggle.com/code/hugoboia/analyzing-pudl-data-with-duckdb
    3. \n
  3. https://www.kaggle.com/code/igjit1/fast-and-less-memory-data-processing-with-duckdb
    4. \n
  4. https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series
    5. \n
  5. https://www.kaggle.com/code/quangtiencs/duckdb-for-feature-engineering-home-credit-2024
    6. \n
  6. https://www.kaggle.com/code/igjit1/amex-basic-logit-with-tidymodels
    7. \n
  7. https://www.kaggle.com/code/jtbontinck/use-amex-parquet-file-with-duckdb
    8. \n
  8. https://www.kaggle.com/code/thiagomantuani/home-credit-2024-duckdb-sql-introduction
    9. \n
  9. https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive
    10. \n
  10. https://www.kaggle.com/code/ayusov/testing-duckdb
    11. \n
\n

Pyspark

\n
    \n
  1. https://www.kaggle.com/code/fatmakursun/pyspark-ml-tutorial-for-beginners
    2. \n
  2. https://www.kaggle.com/code/tientd95/pyspark-for-data-science
    3. \n
  3. https://www.kaggle.com/code/tientd95/advanced-pyspark-for-exploratory-data-analysis
    4. \n
  4. https://www.kaggle.com/code/ybifoundation/pyspark-for-beginners
    5. \n
  5. https://www.kaggle.com/code/tientd95/pyspark-for-statistical-inference
    6. \n
  6. https://www.kaggle.com/code/akdagmelih/wind-turbine-power-prediction-gbtregressor-pyspark
    7. \n
  7. https://www.kaggle.com/code/nadianizam/h-m-fashion-recommendation-with-pyspark
    8. \n
\n

Polars

\n
    \n
  1. https://www.kaggle.com/code/radek1/polars-proof-of-concept-lgbm-ranker
    2. \n
  2. https://www.kaggle.com/code/roberthatch/lb-1-183-lightning-fast-baseline-with-polars
    3. \n
  3. https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-train
    4. \n
  4. https://www.kaggle.com/code/jetakow/home-credit-2024-starter-notebook
    5. \n
  5. https://www.kaggle.com/code/pourchot/simple-xgb
    6. \n
\n

cuDF

\n
    \n
  1. https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb
    2. \n
  2. https://www.kaggle.com/code/shashwatraman/gpu-xgb-baseline-using-rapids-cudf-train
    3. \n
  3. https://www.kaggle.com/code/confirm/xfeat-cudf-lightgbm-catboost-wip
    4. \n
  4. https://www.kaggle.com/code/alberteinsten/cudf-pandas-proof-of-concept-lgbm-ranker
    5. \n
  5. https://www.kaggle.com/code/utm529fg/otto-tuning-candidate-rerank-model-lb-0-577
    6. \n
\n

I shall update this post with relevant references shortly, this is work in progress!
\nBest wishes!

", "messages": [ { "id": 2760158, "postDate": "2024-04-19T06:21:52.090Z", "content": "

Hello everyone,

\n

Wishing you the best for the upcoming competition! I have handpicked a few resources hoping they will aid you in onboarding for this challenge-

\n

Domain based research papers and reading materials

\n
    \n
  1. https://paperswithcode.com/paper/deepphysinet-bridging-deep-learning-and
    2. \n
  2. https://paperswithcode.com/paper/climax-a-foundation-model-for-weather-and
    3. \n
  3. https://physics.paperswithcode.com/paper/purely-data-driven-medium-range-weather
    4. \n
  4. https://physics.paperswithcode.com/paper/open-source-qgis-toolkit-for-the-advanced
    5. \n
  5. https://paperswithcode.com/paper/spatio-temporal-weather-forecasting-and
    6. \n
  6. https://paperswithcode.com/paper/ace-a-fast-skillful-learned-global
    7. \n
  7. https://paperswithcode.com/paper/climsim-a-large-multi-scale-dataset-for
    8. \n
  8. https://ieeexplore.ieee.org/abstract/document/9139804/
    9. \n
  9. https://ieeexplore.ieee.org/abstract/document/9996169
    10. \n
  10. https://proceedings.neurips.cc/paper_files/paper/2023/hash/45fbcc01349292f5e059a0b8b02c8c3f-Abstract-Datasets_and_Benchmarks.html
    11. \n
  11. https://arxiv.org/abs/2306.08754
    12. \n
\n

This assignment is a tabular dataset with a very large size. 192 GB is a humungous dataset size for one to manipulate. As a word of caution, be mindful of the amount of computational resources available at your end while making a suitable decision to participate in the challenge. Also, kindly note that one may be able to extend the training dataset using the link below. Please peruse the overview page for more details-

\n\n

Manipulating large datasets

\n

DuckDB

\n
    \n
  1. https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest
    2. \n
  2. https://www.kaggle.com/code/hugoboia/analyzing-pudl-data-with-duckdb
    3. \n
  3. https://www.kaggle.com/code/igjit1/fast-and-less-memory-data-processing-with-duckdb
    4. \n
  4. https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series
    5. \n
  5. https://www.kaggle.com/code/quangtiencs/duckdb-for-feature-engineering-home-credit-2024
    6. \n
  6. https://www.kaggle.com/code/igjit1/amex-basic-logit-with-tidymodels
    7. \n
  7. https://www.kaggle.com/code/jtbontinck/use-amex-parquet-file-with-duckdb
    8. \n
  8. https://www.kaggle.com/code/thiagomantuani/home-credit-2024-duckdb-sql-introduction
    9. \n
  9. https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive
    10. \n
  10. https://www.kaggle.com/code/ayusov/testing-duckdb
    11. \n
\n

Pyspark

\n
    \n
  1. https://www.kaggle.com/code/fatmakursun/pyspark-ml-tutorial-for-beginners
    2. \n
  2. https://www.kaggle.com/code/tientd95/pyspark-for-data-science
    3. \n
  3. https://www.kaggle.com/code/tientd95/advanced-pyspark-for-exploratory-data-analysis
    4. \n
  4. https://www.kaggle.com/code/ybifoundation/pyspark-for-beginners
    5. \n
  5. https://www.kaggle.com/code/tientd95/pyspark-for-statistical-inference
    6. \n
  6. https://www.kaggle.com/code/akdagmelih/wind-turbine-power-prediction-gbtregressor-pyspark
    7. \n
  7. https://www.kaggle.com/code/nadianizam/h-m-fashion-recommendation-with-pyspark
    8. \n
\n

Polars

\n
    \n
  1. https://www.kaggle.com/code/radek1/polars-proof-of-concept-lgbm-ranker
    2. \n
  2. https://www.kaggle.com/code/roberthatch/lb-1-183-lightning-fast-baseline-with-polars
    3. \n
  3. https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-train
    4. \n
  4. https://www.kaggle.com/code/jetakow/home-credit-2024-starter-notebook
    5. \n
  5. https://www.kaggle.com/code/pourchot/simple-xgb
    6. \n
\n

cuDF

\n
    \n
  1. https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb
    2. \n
  2. https://www.kaggle.com/code/shashwatraman/gpu-xgb-baseline-using-rapids-cudf-train
    3. \n
  3. https://www.kaggle.com/code/confirm/xfeat-cudf-lightgbm-catboost-wip
    4. \n
  4. https://www.kaggle.com/code/alberteinsten/cudf-pandas-proof-of-concept-lgbm-ranker
    5. \n
  5. https://www.kaggle.com/code/utm529fg/otto-tuning-candidate-rerank-model-lb-0-577
    6. \n
\n

I shall update this post with relevant references shortly, this is work in progress!
\nBest wishes!

", "rawMarkdown": "Hello everyone,\n\nWishing you the best for the upcoming competition! I have handpicked a few resources hoping they will aid you in onboarding for this challenge-\n\n# Domain based research papers and reading materials\n1. https://paperswithcode.com/paper/deepphysinet-bridging-deep-learning-and\n2. https://paperswithcode.com/paper/climax-a-foundation-model-for-weather-and\n3. https://physics.paperswithcode.com/paper/purely-data-driven-medium-range-weather\n4. https://physics.paperswithcode.com/paper/open-source-qgis-toolkit-for-the-advanced\n5. https://paperswithcode.com/paper/spatio-temporal-weather-forecasting-and\n6. https://paperswithcode.com/paper/ace-a-fast-skillful-learned-global\n7. https://paperswithcode.com/paper/climsim-a-large-multi-scale-dataset-for\n8. https://ieeexplore.ieee.org/abstract/document/9139804/\n9. https://ieeexplore.ieee.org/abstract/document/9996169\n10. https://proceedings.neurips.cc/paper_files/paper/2023/hash/45fbcc01349292f5e059a0b8b02c8c3f-Abstract-Datasets_and_Benchmarks.html\n11. https://arxiv.org/abs/2306.08754\n\nThis assignment is a tabular dataset with a very large size. 192 GB is a humungous dataset size for one to manipulate. As a word of caution, be mindful of the amount of computational resources available at your end while making a suitable decision to participate in the challenge. Also, kindly note that one may be able to extend the training dataset using the link below. Please peruse the [overview page](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/overview) for more details- \n- https://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py\n- https://arxiv.org/abs/2306.08754\n\n# Manipulating large datasets\n## DuckDB\n1. https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest\n2. https://www.kaggle.com/code/hugoboia/analyzing-pudl-data-with-duckdb\n3. https://www.kaggle.com/code/igjit1/fast-and-less-memory-data-processing-with-duckdb\n4. https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series\n5. https://www.kaggle.com/code/quangtiencs/duckdb-for-feature-engineering-home-credit-2024\n6. https://www.kaggle.com/code/igjit1/amex-basic-logit-with-tidymodels\n7. https://www.kaggle.com/code/jtbontinck/use-amex-parquet-file-with-duckdb\n8. https://www.kaggle.com/code/thiagomantuani/home-credit-2024-duckdb-sql-introduction\n9. https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive\n10. https://www.kaggle.com/code/ayusov/testing-duckdb\n\n## Pyspark\n1. https://www.kaggle.com/code/fatmakursun/pyspark-ml-tutorial-for-beginners\n2. https://www.kaggle.com/code/tientd95/pyspark-for-data-science\n3. https://www.kaggle.com/code/tientd95/advanced-pyspark-for-exploratory-data-analysis\n4. https://www.kaggle.com/code/ybifoundation/pyspark-for-beginners\n5. https://www.kaggle.com/code/tientd95/pyspark-for-statistical-inference\n6. https://www.kaggle.com/code/akdagmelih/wind-turbine-power-prediction-gbtregressor-pyspark\n7. https://www.kaggle.com/code/nadianizam/h-m-fashion-recommendation-with-pyspark\n\n## Polars\n1. https://www.kaggle.com/code/radek1/polars-proof-of-concept-lgbm-ranker\n2. https://www.kaggle.com/code/roberthatch/lb-1-183-lightning-fast-baseline-with-polars\n3. https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-train\n4. https://www.kaggle.com/code/jetakow/home-credit-2024-starter-notebook\n5. https://www.kaggle.com/code/pourchot/simple-xgb\n\n## cuDF\n1. https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb\n2. https://www.kaggle.com/code/shashwatraman/gpu-xgb-baseline-using-rapids-cudf-train\n3. https://www.kaggle.com/code/confirm/xfeat-cudf-lightgbm-catboost-wip\n4. https://www.kaggle.com/code/alberteinsten/cudf-pandas-proof-of-concept-lgbm-ranker\n5. https://www.kaggle.com/code/utm529fg/otto-tuning-candidate-rerank-model-lb-0-577\n\nI shall update this post with relevant references shortly, this is work in progress!\nBest wishes!\n\n", "votes": 43 }, { "id": 2767291, "postDate": "2024-04-22T08:38:22.440Z", "content": "

hey @ravi20076 can I get some material for machine learning I mean best notebooks for learning? same for NLP and Deep learning. btw thanks for sharing this.

", "rawMarkdown": "hey @ravi20076 can I get some material for machine learning I mean best notebooks for learning? same for NLP and Deep learning. btw thanks for sharing this.", "votes": 1, "replies": [ { "id": 2767815, "postDate": "2024-04-22T14:31:18.497Z", "content": "

@ayushparwal2026 of course my friend, what do you wish to learn? I shall provide based on your objectives

", "rawMarkdown": "@ayushparwal2026 of course my friend, what do you wish to learn? I shall provide based on your objectives", "votes": 1, "replies": [ { "id": 2768016, "postDate": "2024-04-22T16:13:04.843Z", "content": "

actually want to learn machine learning in depth. then deep learning and NLP. can you provide me some material or notebooks?

", "rawMarkdown": "actually want to learn machine learning in depth. then deep learning and NLP. can you provide me some material or notebooks?\n", "votes": 1 }, { "id": 2768393, "postDate": "2024-04-22T20:10:42.320Z", "content": "

@ayushparwal2026 here you go-

\n

NLP

\n
    \n
  1. https://www.kaggle.com/code/abhishek/approaching-almost-any-nlp-problem-on-kaggle
    2. \n
  2. https://www.kaggle.com/code/philculliton/nlp-getting-started-tutorial
    3. \n
  3. https://www.kaggle.com/code/rftexas/nlp-cheatsheet-master-nlp
    4. \n
  4. https://www.kaggle.com/code/matleonard/intro-to-nlp
    5. \n
  5. https://www.kaggle.com/code/gunesevitan/nlp-with-disaster-tweets-eda-cleaning-and-bert
    6. \n
  6. https://www.kaggle.com/code/erikbruin/nlp-on-student-writing-eda
    7. \n
  7. https://www.kaggle.com/code/avikumart/nlp-news-articles-classif-wordembeddings-rnn
    8. \n
\n

General ML

\n
    \n
  1. https://www.kaggle.com/code/dansbecker/your-first-machine-learning-model
    2. \n
  2. https://www.kaggle.com/code/dansbecker/exercise-machine-learning-competitions
    3. \n
  3. https://www.kaggle.com/code/yogeshtak/exercise-your-first-machine-learning-model
    4. \n
  4. https://www.kaggle.com/code/alexisbcook/titanic-tutorial
    5. \n
  5. https://www.kaggle.com/code/arthurtok/introduction-to-ensembling-stacking-in-python
    6. \n
", "rawMarkdown": "@ayushparwal2026 here you go-\n\n## NLP\n1. https://www.kaggle.com/code/abhishek/approaching-almost-any-nlp-problem-on-kaggle\n2. https://www.kaggle.com/code/philculliton/nlp-getting-started-tutorial\n3. https://www.kaggle.com/code/rftexas/nlp-cheatsheet-master-nlp\n4. https://www.kaggle.com/code/matleonard/intro-to-nlp\n5. https://www.kaggle.com/code/gunesevitan/nlp-with-disaster-tweets-eda-cleaning-and-bert\n6. https://www.kaggle.com/code/erikbruin/nlp-on-student-writing-eda\n7. https://www.kaggle.com/code/avikumart/nlp-news-articles-classif-wordembeddings-rnn\n\n## General ML\n1. https://www.kaggle.com/code/dansbecker/your-first-machine-learning-model\n2. https://www.kaggle.com/code/dansbecker/exercise-machine-learning-competitions\n3. https://www.kaggle.com/code/yogeshtak/exercise-your-first-machine-learning-model\n4. https://www.kaggle.com/code/alexisbcook/titanic-tutorial\n5. https://www.kaggle.com/code/arthurtok/introduction-to-ensembling-stacking-in-python", "votes": 2 }, { "id": 2768396, "postDate": "2024-04-22T20:12:55.990Z", "content": "

@ravi20076 thanks you dear sir god bless you. I am doing good or not? you can check my work if possible thanks

", "rawMarkdown": "@ravi20076 thanks you dear sir god bless you. I am doing good or not? you can check my work if possible thanks\n\n", "votes": 1 } ] } ] }, { "id": 2766233, "postDate": "2024-04-21T15:03:57.567Z", "content": "

Thank you for sharing the reference @ravi20076 👍

", "rawMarkdown": "Thank you for sharing the reference @ravi20076 👍", "votes": 1 }, { "id": 2763915, "postDate": "2024-04-20T19:48:46.507Z", "content": "

Thank you for sharing this awesome reference! 😄

", "rawMarkdown": "Thank you for sharing this awesome reference! 😄", "votes": 1 }, { "id": 2761426, "postDate": "2024-04-19T20:20:32.197Z", "content": "

I'm always impressed at how fast you compile this @ravi20076, thanks as always.

", "rawMarkdown": "I'm always impressed at how fast you compile this @ravi20076, thanks as always.", "votes": 1 }, { "id": 2761418, "postDate": "2024-04-19T20:16:50.497Z", "content": "

Good done relevant references

", "rawMarkdown": "Good done relevant references ", "votes": 1 }, { "id": 2761432, "postDate": "2024-04-19T20:22:58.047Z", "content": "

Thanks for great references ! I wanted to learn and work on pyspark and other big data tools. This will be a nice opportunity to flex some muscle on this.. Thanks for all you do in every competition!!🙏

", "rawMarkdown": "Thanks for great references ! I wanted to learn and work on pyspark and other big data tools. This will be a nice opportunity to flex some muscle on this.. Thanks for all you do in every competition!!🙏", "votes": 2 }, { "id": 2771202, "postDate": "2024-04-24T06:33:54.447Z", "content": "

Thanks for sharing the reference !

", "rawMarkdown": "Thanks for sharing the reference !", "votes": 1 }, { "id": 2769844, "postDate": "2024-04-23T15:18:56.317Z", "content": "

Thank you for sharing this @ravi20076

", "rawMarkdown": "Thank you for sharing this @ravi20076 ", "votes": 1 }, { "id": 2768082, "postDate": "2024-04-22T17:10:06.830Z", "content": "

Thank you!

", "rawMarkdown": "Thank you!", "votes": 1 }, { "id": 2767013, "postDate": "2024-04-22T04:27:56.537Z", "content": "

Great efforts 🙌! Thanks for sharing

", "rawMarkdown": "Great efforts 🙌! Thanks for sharing ", "votes": 1 } ], "comments": [ { "id": 2767291, "author_name": "DragonSlayer", "author_url": "", "post_date": "2024-04-22T08:38:22.440000", "content": "

hey @ravi20076 can I get some material for machine learning I mean best notebooks for learning? same for NLP and Deep learning. btw thanks for sharing this.

", "votes": 1, "replies": [ { "id": 2767815, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-04-22T14:31:18.497000", "content": "

@ayushparwal2026 of course my friend, what do you wish to learn? I shall provide based on your objectives

", "votes": 1, "replies": [ { "id": 2768016, "author_name": "DragonSlayer", "author_url": "", "post_date": "2024-04-22T16:13:04.843000", "content": "

actually want to learn machine learning in depth. then deep learning and NLP. can you provide me some material or notebooks?

", "votes": 1, "replies": [] }, { "id": 2768393, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-04-22T20:10:42.320000", "content": "

@ayushparwal2026 here you go-

\n

NLP

\n
    \n
  1. https://www.kaggle.com/code/abhishek/approaching-almost-any-nlp-problem-on-kaggle
    2. \n
  2. https://www.kaggle.com/code/philculliton/nlp-getting-started-tutorial
    3. \n
  3. https://www.kaggle.com/code/rftexas/nlp-cheatsheet-master-nlp
    4. \n
  4. https://www.kaggle.com/code/matleonard/intro-to-nlp
    5. \n
  5. https://www.kaggle.com/code/gunesevitan/nlp-with-disaster-tweets-eda-cleaning-and-bert
    6. \n
  6. https://www.kaggle.com/code/erikbruin/nlp-on-student-writing-eda
    7. \n
  7. https://www.kaggle.com/code/avikumart/nlp-news-articles-classif-wordembeddings-rnn
    8. \n
\n

General ML

\n
    \n
  1. https://www.kaggle.com/code/dansbecker/your-first-machine-learning-model
    2. \n
  2. https://www.kaggle.com/code/dansbecker/exercise-machine-learning-competitions
    3. \n
  3. https://www.kaggle.com/code/yogeshtak/exercise-your-first-machine-learning-model
    4. \n
  4. https://www.kaggle.com/code/alexisbcook/titanic-tutorial
    5. \n
  5. https://www.kaggle.com/code/arthurtok/introduction-to-ensembling-stacking-in-python
    6. \n
", "votes": 2, "replies": [] }, { "id": 2768396, "author_name": "DragonSlayer", "author_url": "", "post_date": "2024-04-22T20:12:55.990000", "content": "

@ravi20076 thanks you dear sir god bless you. I am doing good or not? you can check my work if possible thanks

", "votes": 1, "replies": [] } ] } ] }, { "id": 2766233, "author_name": "Satej Raste", "author_url": "", "post_date": "2024-04-21T15:03:57.567000", "content": "

Thank you for sharing the reference @ravi20076 👍

", "votes": 1, "replies": [] }, { "id": 2763915, "author_name": "Aaron Lee", "author_url": "", "post_date": "2024-04-20T19:48:46.507000", "content": "

Thank you for sharing this awesome reference! 😄

", "votes": 1, "replies": [] }, { "id": 2761426, "author_name": "asarvazyan", "author_url": "", "post_date": "2024-04-19T20:20:32.197000", "content": "

I'm always impressed at how fast you compile this @ravi20076, thanks as always.

", "votes": 1, "replies": [] }, { "id": 2761418, "author_name": "", "author_url": "", "post_date": "2024-04-19T20:16:50.497000", "content": "

Good done relevant references

", "votes": 1, "replies": [] }, { "id": 2761432, "author_name": "Chirag Desai", "author_url": "", "post_date": "2024-04-19T20:22:58.047000", "content": "

Thanks for great references ! I wanted to learn and work on pyspark and other big data tools. This will be a nice opportunity to flex some muscle on this.. Thanks for all you do in every competition!!🙏

", "votes": 2, "replies": [] }, { "id": 2771202, "author_name": "Hina Ismail", "author_url": "", "post_date": "2024-04-24T06:33:54.447000", "content": "

Thanks for sharing the reference !

", "votes": 1, "replies": [] }, { "id": 2769844, "author_name": "Akshay Hebbar", "author_url": "", "post_date": "2024-04-23T15:18:56.317000", "content": "

Thank you for sharing this @ravi20076

", "votes": 1, "replies": [] }, { "id": 2768082, "author_name": "Bohdan Spesyvtsev", "author_url": "", "post_date": "2024-04-22T17:10:06.830000", "content": "

Thank you!

", "votes": 1, "replies": [] }, { "id": 2767013, "author_name": "Abhishek Biradar", "author_url": "", "post_date": "2024-04-22T04:27:56.537000", "content": "

Great efforts 🙌! Thanks for sharing

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2760158": "Hello everyone,\n\nWishing you the best for the upcoming competition! I have handpicked a few resources hoping they will aid you in onboarding for this challenge-\n\n# Domain based research papers and reading materials\n1. https://paperswithcode.com/paper/deepphysinet-bridging-deep-learning-and\n2. https://paperswithcode.com/paper/climax-a-foundation-model-for-weather-and\n3. https://physics.paperswithcode.com/paper/purely-data-driven-medium-range-weather\n4. https://physics.paperswithcode.com/paper/open-source-qgis-toolkit-for-the-advanced\n5. https://paperswithcode.com/paper/spatio-temporal-weather-forecasting-and\n6. https://paperswithcode.com/paper/ace-a-fast-skillful-learned-global\n7. https://paperswithcode.com/paper/climsim-a-large-multi-scale-dataset-for\n8. https://ieeexplore.ieee.org/abstract/document/9139804/\n9. https://ieeexplore.ieee.org/abstract/document/9996169\n10. https://proceedings.neurips.cc/paper_files/paper/2023/hash/45fbcc01349292f5e059a0b8b02c8c3f-Abstract-Datasets_and_Benchmarks.html\n11. https://arxiv.org/abs/2306.08754\n\nThis assignment is a tabular dataset with a very large size. 192 GB is a humungous dataset size for one to manipulate. As a word of caution, be mindful of the amount of computational resources available at your end while making a suitable decision to participate in the challenge. Also, kindly note that one may be able to extend the training dataset using the link below. Please peruse the [overview page](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/overview) for more details- \n- https://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py\n- https://arxiv.org/abs/2306.08754\n\n# Manipulating large datasets\n## DuckDB\n1. https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest\n2. https://www.kaggle.com/code/hugoboia/analyzing-pudl-data-with-duckdb\n3. https://www.kaggle.com/code/igjit1/fast-and-less-memory-data-processing-with-duckdb\n4. https://www.kaggle.com/code/gvyshnya/plotting-big-data-sleeping-series\n5. https://www.kaggle.com/code/quangtiencs/duckdb-for-feature-engineering-home-credit-2024\n6. https://www.kaggle.com/code/igjit1/amex-basic-logit-with-tidymodels\n7. https://www.kaggle.com/code/jtbontinck/use-amex-parquet-file-with-duckdb\n8. https://www.kaggle.com/code/thiagomantuani/home-credit-2024-duckdb-sql-introduction\n9. https://www.kaggle.com/code/gvyshnya/to-sleep-or-not-to-sleep-deep-eda-dive\n10. https://www.kaggle.com/code/ayusov/testing-duckdb\n\n## Pyspark\n1. https://www.kaggle.com/code/fatmakursun/pyspark-ml-tutorial-for-beginners\n2. https://www.kaggle.com/code/tientd95/pyspark-for-data-science\n3. https://www.kaggle.com/code/tientd95/advanced-pyspark-for-exploratory-data-analysis\n4. https://www.kaggle.com/code/ybifoundation/pyspark-for-beginners\n5. https://www.kaggle.com/code/tientd95/pyspark-for-statistical-inference\n6. https://www.kaggle.com/code/akdagmelih/wind-turbine-power-prediction-gbtregressor-pyspark\n7. https://www.kaggle.com/code/nadianizam/h-m-fashion-recommendation-with-pyspark\n\n## Polars\n1. https://www.kaggle.com/code/radek1/polars-proof-of-concept-lgbm-ranker\n2. https://www.kaggle.com/code/roberthatch/lb-1-183-lightning-fast-baseline-with-polars\n3. https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-train\n4. https://www.kaggle.com/code/jetakow/home-credit-2024-starter-notebook\n5. https://www.kaggle.com/code/pourchot/simple-xgb\n\n## cuDF\n1. https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb\n2. https://www.kaggle.com/code/shashwatraman/gpu-xgb-baseline-using-rapids-cudf-train\n3. https://www.kaggle.com/code/confirm/xfeat-cudf-lightgbm-catboost-wip\n4. https://www.kaggle.com/code/alberteinsten/cudf-pandas-proof-of-concept-lgbm-ranker\n5. https://www.kaggle.com/code/utm529fg/otto-tuning-candidate-rerank-model-lb-0-577\n\nI shall update this post with relevant references shortly, this is work in progress!\nBest wishes!\n\n", "2767291": "hey @ravi20076 can I get some material for machine learning I mean best notebooks for learning? same for NLP and Deep learning. btw thanks for sharing this.", "2766233": "Thank you for sharing the reference @ravi20076 👍", "2763915": "Thank you for sharing this awesome reference! 😄", "2761426": "I'm always impressed at how fast you compile this @ravi20076, thanks as always.", "2761418": "Good done relevant references ", "2761432": "Thanks for great references ! I wanted to learn and work on pyspark and other big data tools. This will be a nice opportunity to flex some muscle on this.. Thanks for all you do in every competition!!🙏", "2771202": "Thanks for sharing the reference !", "2769844": "Thank you for sharing this @ravi20076 ", "2768082": "Thank you!", "2767013": "Great efforts 🙌! Thanks for sharing " } }