{ "id": 409439, "title": "Can someone demonstrate how to handle & play around with datasets of such massive scales through a Notebook? ", "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/409439", "author_name": "Aryan Garg", "post_date": "2023-05-11T03:09:28.616000", "votes": 4, "comment_count": 6, "views": 0, "content": "

Also, do you ever want to compute statistics for the dataset? (~450 GBs for this one ooof)

", "messages": [ { "id": 2254478, "postDate": "2023-05-11T03:09:28.617Z", "content": "

Also, do you ever want to compute statistics for the dataset? (~450 GBs for this one ooof)

", "rawMarkdown": "Also, do you ever want to compute statistics for the dataset? (~450 GBs for this one ooof)", "votes": 3 }, { "id": 2255450, "postDate": "2023-05-11T18:07:38.747Z", "content": "

I wrote a pytorch dataloader that might interest you:
\nhttps://www.kaggle.com/code/thomasrochefort/pytorch-dataloader-example

\n

It allows for fetching batches from the dataset without loading everything in memory.

", "rawMarkdown": "I wrote a pytorch dataloader that might interest you:\nhttps://www.kaggle.com/code/thomasrochefort/pytorch-dataloader-example\n\nIt allows for fetching batches from the dataset without loading everything in memory.", "votes": 1, "replies": [ { "id": 2257102, "postDate": "2023-05-13T03:29:12.877Z", "content": "

Hey Thomas, thanks for the example. I ran your NB but the RAM is overloaded on a simple loop through the dataloader :(

\n

Is there any workaround?

", "rawMarkdown": "Hey Thomas, thanks for the example. I ran your NB but the RAM is overloaded on a simple loop through the dataloader :(\n\nIs there any workaround?", "replies": [ { "id": 2259061, "postDate": "2023-05-14T16:58:20.663Z", "content": "

You have to load the data batchwise, which means you have to overwrite old batches. The dataloader should work fine in every NB

", "rawMarkdown": "You have to load the data batchwise, which means you have to overwrite old batches. The dataloader should work fine in every NB" }, { "id": 2259149, "postDate": "2023-05-14T18:04:22.807Z", "content": "

I too was having the same issue. Uncommented the loop you wrote to load data, but its too large to handle

", "rawMarkdown": "I too was having the same issue. Uncommented the loop you wrote to load data, but its too large to handle" }, { "id": 2261696, "postDate": "2023-05-16T14:02:00.097Z", "content": "

Hello! The dataloader is meant to be used in a pytorch training loop which will load batches and overwrite old ones. Of course if you keep in memory every batch the RAM will explode. Do you have a code snippet to show how you use it?

", "rawMarkdown": "Hello! The dataloader is meant to be used in a pytorch training loop which will load batches and overwrite old ones. Of course if you keep in memory every batch the RAM will explode. Do you have a code snippet to show how you use it?" }, { "id": 2261887, "postDate": "2023-05-16T15:52:07.693Z", "content": "

I'm using it like this here: DeeplabV3+ Submission NB

\n

Let me know if I'm doing it wrong or if there can be optimizations. Thanks in advance :)

", "rawMarkdown": "I'm using it like this here: [DeeplabV3+ Submission NB](https://www.kaggle.com/code/aryangarg01/submission-deeplabv3-on-ash-images-for-contrails)\n\nLet me know if I'm doing it wrong or if there can be optimizations. Thanks in advance :)" } ] } ] } ], "comments": [ { "id": 2255450, "author_name": "Thomas Rochefort-Beaudoin", "author_url": "", "post_date": "2023-05-11T18:07:38.747000", "content": "

I wrote a pytorch dataloader that might interest you:
\nhttps://www.kaggle.com/code/thomasrochefort/pytorch-dataloader-example

\n

It allows for fetching batches from the dataset without loading everything in memory.

", "votes": 1, "replies": [ { "id": 2257102, "author_name": "Aryan Garg", "author_url": "", "post_date": "2023-05-13T03:29:12.877000", "content": "

Hey Thomas, thanks for the example. I ran your NB but the RAM is overloaded on a simple loop through the dataloader :(

\n

Is there any workaround?

", "votes": 0, "replies": [ { "id": 2259061, "author_name": "Patchef", "author_url": "", "post_date": "2023-05-14T16:58:20.663000", "content": "

You have to load the data batchwise, which means you have to overwrite old batches. The dataloader should work fine in every NB

", "votes": 0, "replies": [] }, { "id": 2259149, "author_name": "Proteek Chaudhuri", "author_url": "", "post_date": "2023-05-14T18:04:22.807000", "content": "

I too was having the same issue. Uncommented the loop you wrote to load data, but its too large to handle

", "votes": 0, "replies": [] }, { "id": 2261696, "author_name": "Thomas Rochefort-Beaudoin", "author_url": "", "post_date": "2023-05-16T14:02:00.097000", "content": "

Hello! The dataloader is meant to be used in a pytorch training loop which will load batches and overwrite old ones. Of course if you keep in memory every batch the RAM will explode. Do you have a code snippet to show how you use it?

", "votes": 0, "replies": [] }, { "id": 2261887, "author_name": "Aryan Garg", "author_url": "", "post_date": "2023-05-16T15:52:07.693000", "content": "

I'm using it like this here: DeeplabV3+ Submission NB

\n

Let me know if I'm doing it wrong or if there can be optimizations. Thanks in advance :)

", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2254478": "Also, do you ever want to compute statistics for the dataset? (~450 GBs for this one ooof)", "2255450": "I wrote a pytorch dataloader that might interest you:\nhttps://www.kaggle.com/code/thomasrochefort/pytorch-dataloader-example\n\nIt allows for fetching batches from the dataset without loading everything in memory." } }