{ "id": 358444, "title": "Understanding the Data (Gravitational Waves & STFs)", "url": "/competitions/g2net-detecting-continuous-gravitational-waves/discussion/358444", "author_name": "Ravi Shah", "post_date": "2022-10-07T22:22:40.572000", "votes": 134, "comment_count": 16, "views": 0, "content": "

Data Structure

\n

Here is the structure for each hdf5 file
\n\"\"

\n\n

Gravitational Waves

\n

Competition Goal

\n\n

Background

\n\n

Continuous Gravitational Waves

\n

Here are some important mentioned points from https://www.ligo.org/science/GW-Continuous.php

\n

A continuous gravitational-wave signal from a Galactic neutron star will look almost perfectly constant in both frequency and amplitude.
\n\"\"

\n

However, over longer durations, the frequency of the signal will slowly change, for two reasons

\n
    \n
  1. as the neutron star emits gravitational and electromagnetic waves, it loses energy which causes it to rotate more slowly
    2. \n
  2. the detector here on Earth is moving with respect to the neutron star, which changes the frequency of the gravitational waves observed in the detector
    3. \n
\n

Tracking all possible frequency changes is what makes the detection of continuous gravitational waves a computational challenge.
\n\"\"

\n

Short-time Fourier Transforms (SFTs)

\n\n

computing stft
\n\"\"
\nsource: https://en.wikipedia.org/wiki/Short-time_Fourier_transform

\n

References

\n

competition data description
\nreading the hdf5 files by @chazzer
\ngetting started notebook by @edwardcrookenden
\nThe new frontier of gravitational waves Research Paper by Miller et al.
\ninfo on Continuous Gravitational Waves by LIGO
\nscipy docs on stft by scipy
\nwiki on stft

", "messages": [ { "id": 1977247, "postDate": "2022-10-07T22:22:40.573Z", "content": "

Data Structure

\n

Here is the structure for each hdf5 file
\n\"\"

\n\n

Gravitational Waves

\n

Competition Goal

\n\n

Background

\n\n

Continuous Gravitational Waves

\n

Here are some important mentioned points from https://www.ligo.org/science/GW-Continuous.php

\n

A continuous gravitational-wave signal from a Galactic neutron star will look almost perfectly constant in both frequency and amplitude.
\n\"\"

\n

However, over longer durations, the frequency of the signal will slowly change, for two reasons

\n
    \n
  1. as the neutron star emits gravitational and electromagnetic waves, it loses energy which causes it to rotate more slowly
    2. \n
  2. the detector here on Earth is moving with respect to the neutron star, which changes the frequency of the gravitational waves observed in the detector
    3. \n
\n

Tracking all possible frequency changes is what makes the detection of continuous gravitational waves a computational challenge.
\n\"\"

\n

Short-time Fourier Transforms (SFTs)

\n\n

computing stft
\n\"\"
\nsource: https://en.wikipedia.org/wiki/Short-time_Fourier_transform

\n

References

\n

competition data description
\nreading the hdf5 files by @chazzer
\ngetting started notebook by @edwardcrookenden
\nThe new frontier of gravitational waves Research Paper by Miller et al.
\ninfo on Continuous Gravitational Waves by LIGO
\nscipy docs on stft by scipy
\nwiki on stft

", "rawMarkdown": "# Data Structure\nHere is the structure for each hdf5 file\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2F1893bf6ae0ffb282512eec0fdce5a662%2Fstruc.PNG?generation=1665177635295503&alt=media)\n- L1 - data from the LIGO Livingston interferometer\n- H1 - data from the LIGO Hanford interferometer\n- STFs - Short-time Fourier Transforms (SFTs) - shape (360, n)\n- timestamps - the timestamps that the STFs correspond to - shape (n,)\n- frequency Hz - the range frequencies measured by the detectors - shape (360,)\n\n# Gravitational Waves \n\n### Competition Goal\n- The goal of this competition is to create a model that can detect continuous gravitational-wave signals. \n- These are weak yet long-lasting signals emitted by rapidly-spinning neutron stars within noisy data. \n- Target is 1 if signal is present else 0\n\n### Background\n- Gravitational Waves are disturbances or ripples in the curvature of spacetime, generated by accelerated masses\n- Gravitational Waves were first proposed by Einstein in 1916 before being directly observed a century later in 2015\n- LIGO (notice this is where our data comes from) is a detector in Livingston, Louisiana and Hanford, Washington. The detector has a L shape with equal-length arms connected to a corner station. When a gravitational wave passes by, one leg of the detector is shortened while the other is lengthened. The interference makes a shift that we can analyze.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2Ff64e191975a1ebc2b76f6648da5eec95%2Fgrav%20wave.png?generation=1665179252370254&alt=media)\nsource: https://www.nature.com/articles/s41586-019-1129-z\n\n### Continuous Gravitational Waves\nHere are some important mentioned points from https://www.ligo.org/science/GW-Continuous.php\n\nA continuous gravitational-wave signal from a Galactic neutron star will look almost perfectly constant in both frequency and amplitude.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2F6b476a7c1f076bdf7c4f7234d37ace3b%2Fperfect%20cont%20wave.PNG?generation=1665181150773656&alt=media)\n\nHowever, over longer durations, the frequency of the signal will slowly change, for two reasons\n1. as the neutron star emits gravitational and electromagnetic waves, it loses energy which causes it to rotate more slowly\n2. the detector here on Earth is moving with respect to the neutron star, which changes the frequency of the gravitational waves observed in the detector\n\nTracking all possible frequency changes is what makes the detection of continuous gravitational waves a computational challenge.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2F5c1472d7cb7ed505784dbbf5ea5f76ba%2Fcont%20grav%20wave.PNG?generation=1665181055200776&alt=media)\n\n# Short-time Fourier Transforms (SFTs)\n- Short-time Fourier Transforms (SFTs) can be used as a way of quantifying the change of a nonstationary signal’s frequency and phase content over time.\n- This is a very common type of signal preprocessing technique\n- This is the type of data we were given to work with in the hdf5 files\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2Fa8a7caafee2261f27fde68045a824202%2Fstft.PNG?generation=1665180281946268&alt=media)\nsource: https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.stft.html\n\n**computing stft**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2Ff251c781289a238b183278fd0b4335bb%2Fwiki%20stft.PNG?generation=1665180491180184&alt=media)\nsource: https://en.wikipedia.org/wiki/Short-time_Fourier_transform\n\n\n# References\n[competition data description](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/data)\n[reading the hdf5 files](https://www.kaggle.com/code/chazzer/how-to-read-the-hdf5-files) by @chazzer\n[getting started notebook](https://www.kaggle.com/code/edwardcrookenden/g2net-getting-started-eda/notebook) by @edwardcrookenden\n[The new frontier of gravitational waves Research Paper](https://www.nature.com/articles/s41586-019-1129-z) by Miller et al.\n[info on Continuous Gravitational Waves](https://www.ligo.org/science/GW-Continuous.php) by LIGO\n[scipy docs on stft](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.stft.html) by scipy\n[wiki on stft](https://en.wikipedia.org/wiki/Short-time_Fourier_transform)", "votes": 134 }, { "id": 2010964, "postDate": "2022-10-31T09:28:29.863Z", "content": "

Great explanation with a lot of great schemas, thanks for sharing!

", "rawMarkdown": "Great explanation with a lot of great schemas, thanks for sharing!", "votes": 1 }, { "id": 1982818, "postDate": "2022-10-11T16:37:02.030Z", "content": "

This is great walk-through @ravishah1. Thanks for sharing.

", "rawMarkdown": "This is great walk-through @ravishah1. Thanks for sharing.", "votes": 1 }, { "id": 1977332, "postDate": "2022-10-08T00:34:44.780Z", "content": "

Wow, great summary!

", "rawMarkdown": "Wow, great summary!", "votes": 1 }, { "id": 2058444, "postDate": "2022-12-08T00:23:07.197Z", "content": "

Thanks for sharing! it's easy to make sense. especially, data structure image is very simple and nice infomation.

", "rawMarkdown": "Thanks for sharing! it's easy to make sense. especially, data structure image is very simple and nice infomation. ", "votes": 2 }, { "id": 2006849, "postDate": "2022-10-27T20:46:48.377Z", "content": "

Great article .. quite informative for beginners in this field. thanks for sharing this.

", "rawMarkdown": "Great article .. quite informative for beginners in this field. thanks for sharing this. ", "votes": 2 }, { "id": 1982195, "postDate": "2022-10-11T10:18:10.570Z", "content": "

The waves are volatile, so it needs to be filtered.

", "rawMarkdown": "The waves are volatile, so it needs to be filtered.", "votes": 2 }, { "id": 1981011, "postDate": "2022-10-10T14:05:45.343Z", "content": "

Amazing explanation. Thanks for putting this together.

", "rawMarkdown": "Amazing explanation. Thanks for putting this together.", "votes": 2 }, { "id": 1978844, "postDate": "2022-10-09T03:39:15.693Z", "content": "

Very organized and well put together, great work!

", "rawMarkdown": "Very organized and well put together, great work!", "votes": 2 }, { "id": 2068987, "postDate": "2022-12-18T13:29:47.310Z", "content": "

This discussion has helped me a lot, thank you.

", "rawMarkdown": "This discussion has helped me a lot, thank you." }, { "id": 2071602, "postDate": "2022-12-21T08:08:54.323Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2001493, "postDate": "2022-10-24T05:00:55.563Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1979159, "postDate": "2022-10-09T07:22:54.580Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1983295, "postDate": "2022-10-12T00:12:11.757Z", "content": "

Thanks for sharing this @ravishah1

", "rawMarkdown": "Thanks for sharing this @ravishah1 ", "votes": 1 }, { "id": 1982767, "postDate": "2022-10-11T16:08:49.607Z", "content": "

Thanks for the overview.

", "rawMarkdown": "Thanks for the overview.", "votes": 2 }, { "id": 2001182, "postDate": "2022-10-23T20:13:46.333Z", "content": "

Great research. Thanks!

", "rawMarkdown": "Great research. Thanks!" }, { "id": 2066965, "postDate": "2022-12-16T08:38:40.317Z", "content": "

Thanks for your sharing!

", "rawMarkdown": "Thanks for your sharing!" } ], "comments": [ { "id": 2010964, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2022-10-31T09:28:29.863000", "content": "

Great explanation with a lot of great schemas, thanks for sharing!

", "votes": 1, "replies": [] }, { "id": 1982818, "author_name": "Oscar Aguilar", "author_url": "", "post_date": "2022-10-11T16:37:02.030000", "content": "

This is great walk-through @ravishah1. Thanks for sharing.

", "votes": 1, "replies": [] }, { "id": 1977332, "author_name": "Aaditya Agnihotri", "author_url": "", "post_date": "2022-10-08T00:34:44.780000", "content": "

Wow, great summary!

", "votes": 1, "replies": [] }, { "id": 2058444, "author_name": "shun takinami", "author_url": "", "post_date": "2022-12-08T00:23:07.197000", "content": "

Thanks for sharing! it's easy to make sense. especially, data structure image is very simple and nice infomation.

", "votes": 2, "replies": [] }, { "id": 2006849, "author_name": "Chirag Desai", "author_url": "", "post_date": "2022-10-27T20:46:48.377000", "content": "

Great article .. quite informative for beginners in this field. thanks for sharing this.

", "votes": 2, "replies": [] }, { "id": 1982195, "author_name": "柯慕灵", "author_url": "", "post_date": "2022-10-11T10:18:10.570000", "content": "

The waves are volatile, so it needs to be filtered.

", "votes": 2, "replies": [] }, { "id": 1981011, "author_name": "Jared Savage", "author_url": "", "post_date": "2022-10-10T14:05:45.343000", "content": "

Amazing explanation. Thanks for putting this together.

", "votes": 2, "replies": [] }, { "id": 1978844, "author_name": "Will", "author_url": "", "post_date": "2022-10-09T03:39:15.693000", "content": "

Very organized and well put together, great work!

", "votes": 2, "replies": [] }, { "id": 2068987, "author_name": "berber", "author_url": "", "post_date": "2022-12-18T13:29:47.310000", "content": "

This discussion has helped me a lot, thank you.

", "votes": 0, "replies": [] }, { "id": 2071602, "author_name": "", "author_url": "", "post_date": "2022-12-21T08:08:54.323000", "content": "", "votes": 0, "replies": [] }, { "id": 2001493, "author_name": "", "author_url": "", "post_date": "2022-10-24T05:00:55.563000", "content": "", "votes": 1, "replies": [] }, { "id": 1979159, "author_name": "", "author_url": "", "post_date": "2022-10-09T07:22:54.580000", "content": "", "votes": 1, "replies": [] }, { "id": 1983295, "author_name": "Godsent Abode", "author_url": "", "post_date": "2022-10-12T00:12:11.757000", "content": "

Thanks for sharing this @ravishah1

", "votes": 1, "replies": [] }, { "id": 1982767, "author_name": "Lester Jones", "author_url": "", "post_date": "2022-10-11T16:08:49.607000", "content": "

Thanks for the overview.

", "votes": 2, "replies": [] }, { "id": 2001182, "author_name": "aspiring", "author_url": "", "post_date": "2022-10-23T20:13:46.333000", "content": "

Great research. Thanks!

", "votes": 0, "replies": [] }, { "id": 2066965, "author_name": "Ziwei Guo", "author_url": "", "post_date": "2022-12-16T08:38:40.317000", "content": "

Thanks for your sharing!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1977247": "# Data Structure\nHere is the structure for each hdf5 file\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2F1893bf6ae0ffb282512eec0fdce5a662%2Fstruc.PNG?generation=1665177635295503&alt=media)\n- L1 - data from the LIGO Livingston interferometer\n- H1 - data from the LIGO Hanford interferometer\n- STFs - Short-time Fourier Transforms (SFTs) - shape (360, n)\n- timestamps - the timestamps that the STFs correspond to - shape (n,)\n- frequency Hz - the range frequencies measured by the detectors - shape (360,)\n\n# Gravitational Waves \n\n### Competition Goal\n- The goal of this competition is to create a model that can detect continuous gravitational-wave signals. \n- These are weak yet long-lasting signals emitted by rapidly-spinning neutron stars within noisy data. \n- Target is 1 if signal is present else 0\n\n### Background\n- Gravitational Waves are disturbances or ripples in the curvature of spacetime, generated by accelerated masses\n- Gravitational Waves were first proposed by Einstein in 1916 before being directly observed a century later in 2015\n- LIGO (notice this is where our data comes from) is a detector in Livingston, Louisiana and Hanford, Washington. The detector has a L shape with equal-length arms connected to a corner station. When a gravitational wave passes by, one leg of the detector is shortened while the other is lengthened. The interference makes a shift that we can analyze.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2Ff64e191975a1ebc2b76f6648da5eec95%2Fgrav%20wave.png?generation=1665179252370254&alt=media)\nsource: https://www.nature.com/articles/s41586-019-1129-z\n\n### Continuous Gravitational Waves\nHere are some important mentioned points from https://www.ligo.org/science/GW-Continuous.php\n\nA continuous gravitational-wave signal from a Galactic neutron star will look almost perfectly constant in both frequency and amplitude.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2F6b476a7c1f076bdf7c4f7234d37ace3b%2Fperfect%20cont%20wave.PNG?generation=1665181150773656&alt=media)\n\nHowever, over longer durations, the frequency of the signal will slowly change, for two reasons\n1. as the neutron star emits gravitational and electromagnetic waves, it loses energy which causes it to rotate more slowly\n2. the detector here on Earth is moving with respect to the neutron star, which changes the frequency of the gravitational waves observed in the detector\n\nTracking all possible frequency changes is what makes the detection of continuous gravitational waves a computational challenge.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2F5c1472d7cb7ed505784dbbf5ea5f76ba%2Fcont%20grav%20wave.PNG?generation=1665181055200776&alt=media)\n\n# Short-time Fourier Transforms (SFTs)\n- Short-time Fourier Transforms (SFTs) can be used as a way of quantifying the change of a nonstationary signal’s frequency and phase content over time.\n- This is a very common type of signal preprocessing technique\n- This is the type of data we were given to work with in the hdf5 files\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2Fa8a7caafee2261f27fde68045a824202%2Fstft.PNG?generation=1665180281946268&alt=media)\nsource: https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.stft.html\n\n**computing stft**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2Ff251c781289a238b183278fd0b4335bb%2Fwiki%20stft.PNG?generation=1665180491180184&alt=media)\nsource: https://en.wikipedia.org/wiki/Short-time_Fourier_transform\n\n\n# References\n[competition data description](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/data)\n[reading the hdf5 files](https://www.kaggle.com/code/chazzer/how-to-read-the-hdf5-files) by @chazzer\n[getting started notebook](https://www.kaggle.com/code/edwardcrookenden/g2net-getting-started-eda/notebook) by @edwardcrookenden\n[The new frontier of gravitational waves Research Paper](https://www.nature.com/articles/s41586-019-1129-z) by Miller et al.\n[info on Continuous Gravitational Waves](https://www.ligo.org/science/GW-Continuous.php) by LIGO\n[scipy docs on stft](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.stft.html) by scipy\n[wiki on stft](https://en.wikipedia.org/wiki/Short-time_Fourier_transform)", "2010964": "Great explanation with a lot of great schemas, thanks for sharing!", "1982818": "This is great walk-through @ravishah1. Thanks for sharing.", "1977332": "Wow, great summary!", "2058444": "Thanks for sharing! it's easy to make sense. especially, data structure image is very simple and nice infomation. ", "2006849": "Great article .. quite informative for beginners in this field. thanks for sharing this. ", "1982195": "The waves are volatile, so it needs to be filtered.", "1981011": "Amazing explanation. Thanks for putting this together.", "1978844": "Very organized and well put together, great work!", "2068987": "This discussion has helped me a lot, thank you.", "2071602": "", "2001493": "", "1979159": "", "1983295": "Thanks for sharing this @ravishah1 ", "1982767": "Thanks for the overview.", "2001182": "Great research. Thanks!", "2066965": "Thanks for your sharing!" } }