![\"memory-trick\"](\"https://i.ibb.co/xhndhMQ/memory-trick.png\")
Tried a few ways, most of them failed but there is one way succeeded. Assume that you have unzipped the train_simplified.zip into directory train_simplified/ which contains 340 csv files.
First, merge 340 csv files into a single csv file and only keep two columns, drawing and y, y is an integer converted from the word column, the file size of train_simplfied.csv is 21GB. Then you can just load the whole dataset by df = pd.read_csv('train_simplified.csv'), which will consume around 25GB RAM.
The following are a few failed ways I've tried:
\n\nLoad the train_simplified.csv using df = pd.read_csv('train_simplified.csv', converters={'drawing': json.loads}), this method ended with consuming up all RAM and failed.
Then I optimized the parsing function:
\n\npython\ndef parse_strokes(s):\nx = json.loads(s)\nreturn list(map(np.uint8, x))\n
Because all the values in drawing column don't exceed 256 they can be stored in np.uint8, however df = pd.read_csv('train_simplified.csv', converters={'drawing': parse_strokes}) ended with consuming up all RAM and failed again.
Merge 340 csv files into a single hdf5 file, only keep two columns drawing and y, and parse the drawing by either json.loads() or parse_strokes() . After parsing strings to lists, the file size does reduces to 12G, but when I tried to load the dataset into memory, df = pd.read_hdf('train_simplified.hdf5'), the RAM consumption grew over 50GB, since my machine only has 52GB RAM it failed too.
The key takeaway is that do NOT parse the drawing column in advance, instead you should parse them on the fly during training. It seems list of integers consume much more memory than plain strings.
If you store drawing as a string, there is a pretty dumb trick to save about 20% memory without content loss...
... just remove whitespaces 😁
\n\n
Impressive, thanks for sharing! Is there any way to load all data into memory after parsing drawing from string to list of integers?
My PC doesn't have even the half memory of yours, so I did not even try to load everything in memory. :)
\n\nI came up with the common approach to lazily load data. While my GPU is crunching a batch, workers on CPU are loading and generating the images for the next batch. It keeps my GPU busy at 98%.\nThe way I implemented it, is probably less common: I am using SQLite file for data and a list of image index in memory. I gives some details here and source code is on github.
\n\nCurious to know why you want to preload everything. I imagine that you have access to more GPU power than myself. 😁
", "rawMarkdown": "My PC doesn't have even the half memory of yours, so I did not even try to load everything in memory. :)\n\nI came up with the common approach to lazily load data. While my GPU is crunching a batch, workers on CPU are loading and generating the images for the next batch. It keeps my GPU busy at 98%.\nThe way I implemented it, is probably less common: I am using SQLite file for data and a list of image index in memory. I gives some details [here](https://towardsdatascience.com/10-lessons-learned-from-participating-to-google-ai-challenge-268b4aa87efa) and source code is on [github](https://github.com/ebouteillon/kaggle-quickdraw-doodle-recognition-challenge).\n\nCurious to know why you want to preload everything. I imagine that you have access to more GPU power than myself. 😁" }, { "id": 440227, "postDate": "2018-12-17T08:54:25.273Z", "content": "Currently I'm using a method that is in the middle of your way and the way that loads all data to memory, I split training data into 100 hdf5 files, every time I load one HDF5 file into memory, generating batches of images and feed to GPUs, and then load the next HDF5. By doing so I only need to access disk 100 times instead of accessing disk every mini-batch. The fit_generator() didn't enable multiprocessing.
You're correct, I'm going to get a 4-GPU machine, so I'm concerned that CPU will not be fast enough to feed data to GPUs, thus I want to write a data generator that inherit keras.util.Sequence so that I can enable multiprocessing without duplicating data. To write a class inheriting from keras.util.Sequence I feel that this class would be much simpler if it can load the whole data into a single DataFrame.
Tried a few ways, most of them failed but there is one way succeeded. Assume that you have unzipped the train_simplified.zip into directory train_simplified/ which contains 340 csv files.
First, merge 340 csv files into a single csv file and only keep two columns, drawing and y, y is an integer converted from the word column, the file size of train_simplfied.csv is 21GB. Then you can just load the whole dataset by df = pd.read_csv('train_simplified.csv'), which will consume around 25GB RAM.
The following are a few failed ways I've tried:
\n\nLoad the train_simplified.csv using df = pd.read_csv('train_simplified.csv', converters={'drawing': json.loads}), this method ended with consuming up all RAM and failed.
Then I optimized the parsing function:
\n\npython\ndef parse_strokes(s):\nx = json.loads(s)\nreturn list(map(np.uint8, x))\n
Because all the values in drawing column don't exceed 256 they can be stored in np.uint8, however df = pd.read_csv('train_simplified.csv', converters={'drawing': parse_strokes}) ended with consuming up all RAM and failed again.
Merge 340 csv files into a single hdf5 file, only keep two columns drawing and y, and parse the drawing by either json.loads() or parse_strokes() . After parsing strings to lists, the file size does reduces to 12G, but when I tried to load the dataset into memory, df = pd.read_hdf('train_simplified.hdf5'), the RAM consumption grew over 50GB, since my machine only has 52GB RAM it failed too.
The key takeaway is that do NOT parse the drawing column in advance, instead you should parse them on the fly during training. It seems list of integers consume much more memory than plain strings.
If you store drawing as a string, there is a pretty dumb trick to save about 20% memory without content loss...
... just remove whitespaces 😁
\n\n
Impressive, thanks for sharing! Is there any way to load all data into memory after parsing drawing from string to list of integers?
My PC doesn't have even the half memory of yours, so I did not even try to load everything in memory. :)
\n\nI came up with the common approach to lazily load data. While my GPU is crunching a batch, workers on CPU are loading and generating the images for the next batch. It keeps my GPU busy at 98%.\nThe way I implemented it, is probably less common: I am using SQLite file for data and a list of image index in memory. I gives some details here and source code is on github.
\n\nCurious to know why you want to preload everything. I imagine that you have access to more GPU power than myself. 😁
", "votes": 0, "replies": [] }, { "id": 440227, "author_name": "[he.ai]soulmachine", "author_url": "", "post_date": "2018-12-17T08:54:25.273000", "content": "Currently I'm using a method that is in the middle of your way and the way that loads all data to memory, I split training data into 100 hdf5 files, every time I load one HDF5 file into memory, generating batches of images and feed to GPUs, and then load the next HDF5. By doing so I only need to access disk 100 times instead of accessing disk every mini-batch. The fit_generator() didn't enable multiprocessing.
You're correct, I'm going to get a 4-GPU machine, so I'm concerned that CPU will not be fast enough to feed data to GPUs, thus I want to write a data generator that inherit keras.util.Sequence so that I can enable multiprocessing without duplicating data. To write a class inheriting from keras.util.Sequence I feel that this class would be much simpler if it can load the whole data into a single DataFrame.