{ "id": 357645, "title": "HDF files with indexed dataframes for the training set", "url": "/competitions/g2net-detecting-continuous-gravitational-waves/discussion/357645", "author_name": "JohnM", "post_date": "2022-10-05T06:35:00.403000", "votes": 16, "comment_count": 5, "views": 0, "content": "

You can find processed HDF files for the training set at https://www.kaggle.com/datasets/jpmiller/simplified-dataset. I processed each file so that it contains two datasets, one for Hanford and one for Livingston.

\n

The index on each frame is the frequency array from the original file, and the columns are the timestamps. As you probably know, he Hanford and Livingston readings within a sample can have different timestamp counts.

\n

The files are easily readable by pandas. Here's an example:

\n
filename = \"../input/simplified-dataset/001121a05.h5\"\ndf_h = pd.read_hdf(filename, key='h')\ndf_l = pd.read_hdf(filename, key='l')\n
\n

The dataset description contains code to generate these files.

", "messages": [ { "id": 1972411, "postDate": "2022-10-05T06:35:00.403Z", "content": "

You can find processed HDF files for the training set at https://www.kaggle.com/datasets/jpmiller/simplified-dataset. I processed each file so that it contains two datasets, one for Hanford and one for Livingston.

\n

The index on each frame is the frequency array from the original file, and the columns are the timestamps. As you probably know, he Hanford and Livingston readings within a sample can have different timestamp counts.

\n

The files are easily readable by pandas. Here's an example:

\n
filename = \"../input/simplified-dataset/001121a05.h5\"\ndf_h = pd.read_hdf(filename, key='h')\ndf_l = pd.read_hdf(filename, key='l')\n
\n

The dataset description contains code to generate these files.

", "rawMarkdown": "You can find processed HDF files for the training set at https://www.kaggle.com/datasets/jpmiller/simplified-dataset. I processed each file so that it contains two datasets, one for Hanford and one for Livingston.\n\nThe index on each frame is the frequency array from the original file, and the columns are the timestamps. As you probably know, he Hanford and Livingston readings within a sample can have different timestamp counts.\n\nThe files are easily readable by pandas. Here's an example:\n\n```\nfilename = \"../input/simplified-dataset/001121a05.h5\"\ndf_h = pd.read_hdf(filename, key='h')\ndf_l = pd.read_hdf(filename, key='l')\n```\n\nThe dataset description contains code to generate these files. ", "votes": 15 }, { "id": 1972632, "postDate": "2022-10-05T08:57:07.053Z", "content": "

Thank you ! It's will be easier. @jpmiller do you plan to release the test set as well please in this format? Thank you.

", "rawMarkdown": "Thank you ! It's will be easier. @jpmiller do you plan to release the test set as well please in this format? Thank you.", "replies": [ { "id": 1973262, "postDate": "2022-10-05T14:53:29.993Z", "content": "

I'd like to but haven't figured out a way to stay within the dataset size limits on Kaggle.

", "rawMarkdown": "I'd like to but haven't figured out a way to stay within the dataset size limits on Kaggle. " }, { "id": 1973857, "postDate": "2022-10-05T22:34:21.897Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1972538, "postDate": "2022-10-05T07:44:18.437Z", "content": "

This is likely to be of great help, many thanks and regards @jpmiller

", "rawMarkdown": "This is likely to be of great help, many thanks and regards @jpmiller " }, { "id": 2053196, "postDate": "2022-12-03T00:38:18.330Z", "content": "

Thank you!

", "rawMarkdown": "Thank you! " } ], "comments": [ { "id": 1972632, "author_name": "Defend Intelligence", "author_url": "", "post_date": "2022-10-05T08:57:07.053000", "content": "

Thank you ! It's will be easier. @jpmiller do you plan to release the test set as well please in this format? Thank you.

", "votes": 0, "replies": [ { "id": 1973262, "author_name": "JohnM", "author_url": "", "post_date": "2022-10-05T14:53:29.993000", "content": "

I'd like to but haven't figured out a way to stay within the dataset size limits on Kaggle.

", "votes": 0, "replies": [] }, { "id": 1973857, "author_name": "", "author_url": "", "post_date": "2022-10-05T22:34:21.897000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1972538, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-10-05T07:44:18.437000", "content": "

This is likely to be of great help, many thanks and regards @jpmiller

", "votes": 0, "replies": [] }, { "id": 2053196, "author_name": "Temari", "author_url": "", "post_date": "2022-12-03T00:38:18.330000", "content": "

Thank you!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1972411": "You can find processed HDF files for the training set at https://www.kaggle.com/datasets/jpmiller/simplified-dataset. I processed each file so that it contains two datasets, one for Hanford and one for Livingston.\n\nThe index on each frame is the frequency array from the original file, and the columns are the timestamps. As you probably know, he Hanford and Livingston readings within a sample can have different timestamp counts.\n\nThe files are easily readable by pandas. Here's an example:\n\n```\nfilename = \"../input/simplified-dataset/001121a05.h5\"\ndf_h = pd.read_hdf(filename, key='h')\ndf_l = pd.read_hdf(filename, key='l')\n```\n\nThe dataset description contains code to generate these files. ", "1972632": "Thank you ! It's will be easier. @jpmiller do you plan to release the test set as well please in this format? Thank you.", "1972538": "This is likely to be of great help, many thanks and regards @jpmiller ", "2053196": "Thank you! " } }