I'm having some trouble trying to load all images on my EDA kernel and do some processing, i always get timeout error, anyone know a effective way to do this here using a Kaggle kernel?
", "messages": [ { "id": 405191, "postDate": "2018-10-17T03:19:49.933Z", "content": "Unless you use a custom machine. or something in the cloud you cannot load it all. Kaggle machines have 14 GB of ram this dataset has 27GB, and more 200GB if you want to use the full dataset publicly available online.
\n\nAnother option is maybe use an image generator of some kind that does not load everything at once.
\n\nCheck my kernel in which i used one, did not reach the end yet but supposedly does not load the data until it is needed
", "rawMarkdown": "Unless you use a custom machine. or something in the cloud you cannot load it all. Kaggle machines have 14 GB of ram this dataset has 27GB, and more 200GB if you want to use the full dataset publicly available online. \n\nAnother option is maybe use an image generator of some kind that does not load everything at once.\n\nCheck [my kernel][1] in which i used one, did not reach the end yet but supposedly does not load the data until it is needed\n\n\n [1]: https://www.kaggle.com/amneves/quick-draw-keras-cnn-model", "votes": 1, "replies": [ { "id": 405386, "postDate": "2018-10-17T11:43:13.563Z", "content": "Yes, André, actually i was thinking on something like this, loading the data by batch, in a couple of days i'll try something like this and i'll check your kernel, thanks.
", "rawMarkdown": "Yes, André, actually i was thinking on something like this, loading the data by batch, in a couple of days i'll try something like this and i'll check your kernel, thanks.", "votes": 1 } ] }, { "id": 399896, "postDate": "2018-10-07T03:55:44.687Z", "content": "Since the data is too big (27GB), I don't think the kernel can load all data once.
", "rawMarkdown": "Since the data is too big (27GB), I don't think the kernel can load all data once. ", "votes": 1 }, { "id": 395499, "postDate": "2018-09-28T17:30:35.643Z", "content": "I'm having some trouble trying to load all images on my EDA kernel and do some processing, i always get timeout error, anyone know a effective way to do this here using a Kaggle kernel?
", "rawMarkdown": "I'm having some trouble trying to load all images on my EDA kernel and do some processing, i always get timeout error, anyone know a effective way to do this here using a Kaggle kernel?" } ], "comments": [ { "id": 405191, "author_name": "André Neves", "author_url": "", "post_date": "2018-10-17T03:19:49.933000", "content": "Unless you use a custom machine. or something in the cloud you cannot load it all. Kaggle machines have 14 GB of ram this dataset has 27GB, and more 200GB if you want to use the full dataset publicly available online.
\n\nAnother option is maybe use an image generator of some kind that does not load everything at once.
\n\nCheck my kernel in which i used one, did not reach the end yet but supposedly does not load the data until it is needed
", "votes": 1, "replies": [ { "id": 405386, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2018-10-17T11:43:13.563000", "content": "Yes, André, actually i was thinking on something like this, loading the data by batch, in a couple of days i'll try something like this and i'll check your kernel, thanks.
", "votes": 1, "replies": [] } ] }, { "id": 399896, "author_name": "ChengangLi", "author_url": "", "post_date": "2018-10-07T03:55:44.687000", "content": "Since the data is too big (27GB), I don't think the kernel can load all data once.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "405191": "Unless you use a custom machine. or something in the cloud you cannot load it all. Kaggle machines have 14 GB of ram this dataset has 27GB, and more 200GB if you want to use the full dataset publicly available online. \n\nAnother option is maybe use an image generator of some kind that does not load everything at once.\n\nCheck [my kernel][1] in which i used one, did not reach the end yet but supposedly does not load the data until it is needed\n\n\n [1]: https://www.kaggle.com/amneves/quick-draw-keras-cnn-model", "399896": "Since the data is too big (27GB), I don't think the kernel can load all data once. ", "395499": "I'm having some trouble trying to load all images on my EDA kernel and do some processing, i always get timeout error, anyone know a effective way to do this here using a Kaggle kernel?" } }