{ "id": 359657, "title": "Suggestions on training on the dataset?", "url": "/competitions/g2net-detecting-continuous-gravitational-waves/discussion/359657", "author_name": "Jacob Dawson", "post_date": "2022-10-13T02:09:44.761000", "votes": 6, "comment_count": 2, "views": 0, "content": "

https://www.kaggle.com/code/chazzer/how-to-read-the-hdf5-files has provided some really helpful info on how to extract numpy arrays from the hdf5 format. I tried making my setup and I ran into a problem: how do I bunch these up into a dataset for processing? One way that I've found is that I can process each hdf5 one-by-one like so:

\n

train_dataset = tf.data.Dataset.list_files(train_path+\"*.hdf5\")\nprint(train_dataset.cardinality())\nprint(train_dataset.take(1).get_single_element())

\n

where that last line outputs:
\ntf.Tensor(b'..\\\\g2net-detecting-continuous-gravitational-waves\\\\train\\\\123594dc7.hdf5', shape=(), dtype=string)

\n

I intend to read the hdf5 files one-by-one from these strings using h5py.File(), but I was curious--is there a better way to do this? Would it be better to create a generator/dataset of hdf5 files themselves which can then be retrieved (obviously we can't store all 200 gb in RAM!)?

\n

Thanks!

", "messages": [ { "id": 1984953, "postDate": "2022-10-13T02:09:44.760Z", "content": "

https://www.kaggle.com/code/chazzer/how-to-read-the-hdf5-files has provided some really helpful info on how to extract numpy arrays from the hdf5 format. I tried making my setup and I ran into a problem: how do I bunch these up into a dataset for processing? One way that I've found is that I can process each hdf5 one-by-one like so:

\n

train_dataset = tf.data.Dataset.list_files(train_path+\"*.hdf5\")\nprint(train_dataset.cardinality())\nprint(train_dataset.take(1).get_single_element())

\n

where that last line outputs:
\ntf.Tensor(b'..\\\\g2net-detecting-continuous-gravitational-waves\\\\train\\\\123594dc7.hdf5', shape=(), dtype=string)

\n

I intend to read the hdf5 files one-by-one from these strings using h5py.File(), but I was curious--is there a better way to do this? Would it be better to create a generator/dataset of hdf5 files themselves which can then be retrieved (obviously we can't store all 200 gb in RAM!)?

\n

Thanks!

", "rawMarkdown": "https://www.kaggle.com/code/chazzer/how-to-read-the-hdf5-files has provided some really helpful info on how to extract numpy arrays from the hdf5 format. I tried making my setup and I ran into a problem: how do I bunch these up into a dataset for processing? One way that I've found is that I can process each hdf5 one-by-one like so:\n\n`train_dataset = tf.data.Dataset.list_files(train_path+\"*.hdf5\")\nprint(train_dataset.cardinality())\nprint(train_dataset.take(1).get_single_element())`\n\nwhere that last line outputs:\n`tf.Tensor(b'..\\\\g2net-detecting-continuous-gravitational-waves\\\\train\\\\123594dc7.hdf5', shape=(), dtype=string)`\n\nI intend to read the hdf5 files one-by-one from these strings using h5py.File(), but I was curious--is there a better way to do this? Would it be better to create a generator/dataset of hdf5 files themselves which can then be retrieved (obviously we can't store all 200 gb in RAM!)?\n\nThanks!", "votes": 6 }, { "id": 1985969, "postDate": "2022-10-13T17:58:54.943Z", "content": "

What do you want to do? hdf5 is pretty quick to read.

", "rawMarkdown": "What do you want to do? hdf5 is pretty quick to read. \n", "replies": [ { "id": 1987892, "postDate": "2022-10-14T21:58:56.877Z", "content": "

Perhaps I didn't phrase my thoughts clearly enough. I can see how I'd parse through each hdf5 file individually and feed them to my model, but I don't see how I could make a dataset out of them so that I can run a batch in parallel. Is there a good way to do this in Tensorflow that I'm missing?

", "rawMarkdown": "Perhaps I didn't phrase my thoughts clearly enough. I can see how I'd parse through each hdf5 file individually and feed them to my model, but I don't see how I could make a dataset out of them so that I can run a batch in parallel. Is there a good way to do this in Tensorflow that I'm missing?" } ] } ], "comments": [ { "id": 1985969, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-10-13T17:58:54.943000", "content": "

What do you want to do? hdf5 is pretty quick to read.

", "votes": 0, "replies": [ { "id": 1987892, "author_name": "Jacob Dawson", "author_url": "", "post_date": "2022-10-14T21:58:56.877000", "content": "

Perhaps I didn't phrase my thoughts clearly enough. I can see how I'd parse through each hdf5 file individually and feed them to my model, but I don't see how I could make a dataset out of them so that I can run a batch in parallel. Is there a good way to do this in Tensorflow that I'm missing?

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1984953": "https://www.kaggle.com/code/chazzer/how-to-read-the-hdf5-files has provided some really helpful info on how to extract numpy arrays from the hdf5 format. I tried making my setup and I ran into a problem: how do I bunch these up into a dataset for processing? One way that I've found is that I can process each hdf5 one-by-one like so:\n\n`train_dataset = tf.data.Dataset.list_files(train_path+\"*.hdf5\")\nprint(train_dataset.cardinality())\nprint(train_dataset.take(1).get_single_element())`\n\nwhere that last line outputs:\n`tf.Tensor(b'..\\\\g2net-detecting-continuous-gravitational-waves\\\\train\\\\123594dc7.hdf5', shape=(), dtype=string)`\n\nI intend to read the hdf5 files one-by-one from these strings using h5py.File(), but I was curious--is there a better way to do this? Would it be better to create a generator/dataset of hdf5 files themselves which can then be retrieved (obviously we can't store all 200 gb in RAM!)?\n\nThanks!", "1985969": "What do you want to do? hdf5 is pretty quick to read. \n" } }