![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2F1893bf6ae0ffb282512eec0fdce5a662%2Fstruc.PNG?generation=1665177635295503&alt=media\")
Save yourself some time and get caught up on the latest updates and important discussions that have taken place prior to the competition deadline.
\nThe competition ends in 10 days and there is still time to improve your solution. Here are some of the important topics that have been discussed in the Kaggle forums.
The data is H-U-G-E and comes in hdf5 format, chazzer had written a notebook on how to read the hdf5 files
Shortly after JohnM pushed this one step further and created HDF files with indexed dataframes for the training set. The index on each frame is the frequency array from the original file, and the columns are the timestamps.
Reading Example
\ndf_h = pd.read_hdf(, key = )\naugs the gap (CV-LB) is bigger\"An amazing intro post by Ravi Shah going through all important details about the data of this competition.
\nIn short (check the original for more info)
\n
Competition Goal
\nThe goal of this competition is to create a model that can detect continuous gravitational-wave signals. These are weak yet long-lasting signals emitted by rapidly-spinning neutron stars within noisy data.
\n\n\n\n
\n- Target is 1 if signal is present else 0
\n
LIGO (where the data comes from) is a detector in Livingston, Louisiana and Hanford, Washington. The detector has a L shape with equal-length arms connected to a corner station. When a gravitational wave passes by, one leg of the detector is shortened while the other is lengthened.
\nThe interference makes a shift that we can analyze.
\nWe Get A Continuous Gravitational Waves
\nA continuous gravitational-wave signal from a Galactic neutron star will look almost perfectly constant in both frequency and amplitude.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2F6b476a7c1f076bdf7c4f7234d37ace3b%2Fperfect%20cont%20wave.PNG?generation=1665181150773656&alt=media\")
But over long durations: The frequency of the signal slowly changes, because:
\nShort-time Fourier Transforms (SFTs)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6537187%2Fa8a7caafee2261f27fde68045a824202%2Fstft.PNG?generation=1665180281946268&alt=media\")
There seems to be 2 differences of our data in comparison to conventional SFT's
\nThe time delta between each timestamp is inconsistent: Time between measurement in the SFT's can vary greatly.
The frequency range is consistently ~0.2Hz, BUT the min/max frequency ranges from around ~50to ~498Hz.
We should not interpret all spectrograms as continuous sounds: When taking a look at the timestamps they start at 2009-03-27/28 and end on 2009-07-25/26/27. The recordings are thus gathered during 4 months with often 30 minutes between the recordings.
Important post by 🐢 Jun Koda sharing with us that since the smallest positive number float32 (~ 1e-38) but on the data it is ~1e-22.
\nOnce we square the data, they go beyond this limit.
\n\n\nnp.float32(1.23456e-44) => 1.3e-44.
\n
\n\nAnother great post by (Mark Wijkhuizen)
\n
PyFstat has many parameters but it's documentation however is not very helpful for explaining them.
\nVictor Gonzalez Explained with an incredible comment everything about all the paramaeters.
\n{**pyfstat.injection_parameters.isotropic_amplitude_priors}\n{: {: {: -, : }}, : {: {: -, : }}, : {: {: , : }}}\nMore Parameters
\nQuestion asked by IkaroSilva
\nAnswer (By The Hosts):
\nThe frequency of a signal as seen by a detector a certain time t (what you call it S(t) in your post) does generally depend on the detector, the reason being that different detectors move \"differently\" around the Sun as they are in slightly different positions on Earth.
This distinction may not be that noticeable for some cases if we use the two Advanced LIGO detectors, since they are relatively close to each other, but becomes quite apparent once you include other detectors such as Virgo, which is located in Italy.
A [good] question about the relation beween the batch-size and generalization performance by dragon zhang.
\nShort Answer:
\nFriends dont let friends use minibatches larger than 32.\n[Yann LeCun](https://twitter.com/ylecun/status/989610208497360896).\nAn amazing solutions summary by Sinan Calisir of all winning solutions of the previous competition.
\nWierd Observation: When computing std of spectrogram in complex64, I'm getting almost allways the same value of 1.4973568266036865e-22. If I do the same computation in complex128, the values starts do differ - and they are partly useful as indicator of target label:
\nAnswer (From the Hosts):
\nTL;DR: This is not a data leak, we understand the reason this is happening and it is expected. Nothing to worry about.
\n\n\nand they are partly useful as indicator of target label
\n
Yes, this is expected: stdev is computed from the data, and very strong signals will deviate the value away from the \"noise-only\" value. This should already tell you how \"strong\" signals in the test set are, by the way :) .
\n\n\nAround 80% of test data seems to be affected by this
\n
This is also well understood. We wanted to give as much real data as possible in this challenge, but we didn't have a reliable way of generating more than what the LIGO detectors produced during the run, and we did not want to run into issues about having too much correlation among different samples. For this reason, we decided to generate synthetic data using Gaussian noise, for which a PSD (or ASD, or sqrtSX, they are all essentially equivalent in this context) needs to be specified.
\n\n\nIt would us sidechannel which is not present in real GW search data.
\n
This is not a problem in CW searches. For \"clean\" (relatively quiet) bands we are able to estimate the PSD of the noise with quite good accuracy, and that is sufficient for us to actually perform our analysis.
\nThe dataset description for the competition states:
\n\"Please note the presence of a small number of files labeled -1.\nPhysicists are currently unable to determine the status of these files.\"\nAs it turns out, these are easter eggs!.
\nIf you plot their signal magnitudes as 2D arrays, you'll see:
50f09e37e
\nA cartoon of two black holes about to collide.
![\"\"](\"https://i.ibb.co/NxnJL9M/50f09e37e.jpg\")
62b0dd011
\nImage that was broadcast into the universe in 1974, as part of the SETI program.
\n![\"\"](\"https://i.ibb.co/347VcjZ/62b0dd011.jpg\")
b7666b451
\nThe 2017 physics Nobel Prize winners (for the discovery of gravitational waves): Rainer Weiss, Barry Barish and Kip Thorne.
\n![\"\"](\"https://i.ibb.co/S0NnW3Z/b7666b451.jpg\")
An interesting discussion about methods for reducing the difference between the LB and the CV score.
\nThis competition is very different from the last one. The last contest was the signal of two black holes or neutron stars merging, which is very strong, and implementing a CNN would allow for some level of fit. But continuous gravitational waves are very weak, and even after scientists in the field have implemented various search methods and deep learning tools they have not found any real signal. If you read some of the papers you will see that even the most advanced NNs that have been tried have not reached traditional methods of search in terms of accuracy, and are usually very unsatisfactory after a SNR of less than 10.\n\nI think the huge gap between CV and LB is because 1) the training set is so small with only 600 data, there is so little variation in between that it may be difficult for the network to learn convolution methods in the face of low SNR samples, and 2) I haven't carefully checked the test set, but I think the composition of the test set must be very demanding and there must be a large number of low SNR samples.\n\nIn order to be able to bring CV and LB closer together, I think the most effective way to do this would be to construct a training set that has a large number of samples with a wide range of SNR values, rather than using 600 training samples and then trying various ML tricks.\nAn interesting and informative discussion about the posibility of additional instrumental artifacts (persistent line-like features) to the noise alongside the simulated signals in testing dataset.
\nYes, the artifacts like persistent lines do exist alongside with the CW signal and background noise. I've done some statistical testing of the whole dataset (test+train) and found them. \n\n\nAnother discussion about creating a robust CV strategy.
\nImportant Note by Chenglu
\nWith the generated dataset, I can reduce the gap to 0.1, CV ~0.77 and LB ~0.68, I think this competition is a matter of shrinking the gap, more precisely, it's generating a dataset that matches the distribution of test set.\nThank you for sharing this!!
\nChenglu is sharing with us a common bug that caused some problems (and you need to also be aware of it):
The bug: Normalizing the data this way:
\nnorm_data = data - data.() / (data.() - data.())\nThis is easily influenced by the value of max and min.
\nThe fix:
norm_data = data * 1e22\nThe fixed classifier only achieve 0.66 on classifying generating v.s. testing/training dataset.
\nThe post lists the differences between the two competitions:
\nThe raw data last year are 1d waves, whose x-axis is time, and y-axis is the amplitude of each timestamp. With CQT/CWT, the 1d wave can be converted into a 2d image(spectrum), which x-axis is time, y-axis is frequency and the intensity per unit pixel is the amplitude of a specific frequency within a specific timestamp.
The raw data are 2d images(spectrum), which x-axis is time, y-axis is frequency and the intensity per unit pixel is the amplitude of a specific frequency within a specific timestamp. (Which is the same as the data after CQT last year in format? I am not sure…)
SNR
\nThe SNR in the competition this year is much lower, which means that the signal this year is much weaker, resulting in the auc being much lower.
Important Addition by Alex Z: Physics-wise, last year competition dealt with the waves from compact binary objects, like close orbiting black holes or white dwarfs. This year we face fast spinning asymmetric neutron stars. Last year set-up provides much more explosive waveforms, and much stronger. Continuous GW we are hunting here were never detected so far, so we have a chance to help a real scientific discovery.
\n\nTL;DR: Vladimir Dropped a BOMB.
\n
It might be possible to reverse engineer the injected CW signal by using raw Ligo&Virgo data.
\nIn Short
\n\n\nWow.. well done on this!
\n
Igor Litvin proposed a method with some potential for improving your NN training.
He figered out that aproximately 20% of the gravitation wave spectra are distorted.
And think the distortion was made artificially because of it is exatly the same distortion for H and L shoulders of the interferometer.
It is possible that they multiplied by exactly the same function to H and exactly the same (but different from H) to L shoulder.
Great topic by Martin Kovacevic Buvinic about important insights about the data.
CV: The test data and train data are different: cv gap between validation and test is always large (~0.10). Also low correlation. - BUT! somehow a 20 folds cv model turned out to be much better.
Guess: Using 20 folds is only 5% of the data as validation for each fold, meaning we are only validating with 30 observations and this lead to overfitting, therefore this validation is not good, increasing the number of folds will just lead to overfitting.
Blending: Blending in most of the casses helps.
Data Generation: Adding more data usually improves the generalization of the model, if we check lb it actually improves so adding more data is helpful. The data simulated is the training data, and the best guess is that the key of the competition is to generate data that is similar to the test data.
The Main Insight: Using training data as validation is not optimal, because it is small and different compared to the test set. A good option would be to generate our own data that is similar to the test set and do traininig and validation with that data, also add the training data.
Vladimir Slaykovskiy asks about the easies way to combine generated data into the a single spectrogram.
\nsample = 0.9 * noise + 0.1 + signalAnswer (By The Hosts):
\nIf noise signal are Fourier amplitudes (i.e. the numbers you get get_stft_as_arrays) the generated SFTs have the same timestamps frequency, then yes.\n\nNote that you can literally do whatever you want to that data. For example, you wanted to create an instrumental artifact a circular shape, you could create such a shape a numpy array the appropriate array shape directly added to your data.\n\n`For example it correct to just weight-average them?`\n\nThe standard way which we do this by generating signals the amplitude of which already a certain fraction of your noiseSave yourself some time and get caught up on the latest updates and important discussions that have taken place prior to the competition deadline.
\nThe competition ends in 10 days and there is still time to improve your solution. Here are some of the important topics that have been discussed in the Kaggle forums.
The data is H-U-G-E and comes in hdf5 format, chazzer had written a notebook on how to read the hdf5 files
Shortly after JohnM pushed this one step further and created HDF files with indexed dataframes for the training set. The index on each frame is the frequency array from the original file, and the columns are the timestamps.
Reading Example
\ndf_h = pd.read_hdf(, key = )\naugs the gap (CV-LB) is bigger\"An amazing intro post by Ravi Shah going through all important details about the data of this competition.
\nIn short (check the original for more info)
\n
Competition Goal
\nThe goal of this competition is to create a model that can detect continuous gravitational-wave signals. These are weak yet long-lasting signals emitted by rapidly-spinning neutron stars within noisy data.
\n\n\n\n
\n- Target is 1 if signal is present else 0
\n
LIGO (where the data comes from) is a detector in Livingston, Louisiana and Hanford, Washington. The detector has a L shape with equal-length arms connected to a corner station. When a gravitational wave passes by, one leg of the detector is shortened while the other is lengthened.
\nThe interference makes a shift that we can analyze.
\nWe Get A Continuous Gravitational Waves
\nA continuous gravitational-wave signal from a Galactic neutron star will look almost perfectly constant in both frequency and amplitude.
\n
But over long durations: The frequency of the signal slowly changes, because:
\nShort-time Fourier Transforms (SFTs)
\n
There seems to be 2 differences of our data in comparison to conventional SFT's
\nThe time delta between each timestamp is inconsistent: Time between measurement in the SFT's can vary greatly.
The frequency range is consistently ~0.2Hz, BUT the min/max frequency ranges from around ~50to ~498Hz.
We should not interpret all spectrograms as continuous sounds: When taking a look at the timestamps they start at 2009-03-27/28 and end on 2009-07-25/26/27. The recordings are thus gathered during 4 months with often 30 minutes between the recordings.
Important post by 🐢 Jun Koda sharing with us that since the smallest positive number float32 (~ 1e-38) but on the data it is ~1e-22.
\nOnce we square the data, they go beyond this limit.
\n\n\nnp.float32(1.23456e-44) => 1.3e-44.
\n
\n\nAnother great post by (Mark Wijkhuizen)
\n
PyFstat has many parameters but it's documentation however is not very helpful for explaining them.
\nVictor Gonzalez Explained with an incredible comment everything about all the paramaeters.
\n{**pyfstat.injection_parameters.isotropic_amplitude_priors}\n{: {: {: -, : }}, : {: {: -, : }}, : {: {: , : }}}\nMore Parameters
\nQuestion asked by IkaroSilva
\nAnswer (By The Hosts):
\nThe frequency of a signal as seen by a detector a certain time t (what you call it S(t) in your post) does generally depend on the detector, the reason being that different detectors move \"differently\" around the Sun as they are in slightly different positions on Earth.
This distinction may not be that noticeable for some cases if we use the two Advanced LIGO detectors, since they are relatively close to each other, but becomes quite apparent once you include other detectors such as Virgo, which is located in Italy.
A [good] question about the relation beween the batch-size and generalization performance by dragon zhang.
\nShort Answer:
\nFriends dont let friends use minibatches larger than 32.\n[Yann LeCun](https://twitter.com/ylecun/status/989610208497360896).\nAn amazing solutions summary by Sinan Calisir of all winning solutions of the previous competition.
\nWierd Observation: When computing std of spectrogram in complex64, I'm getting almost allways the same value of 1.4973568266036865e-22. If I do the same computation in complex128, the values starts do differ - and they are partly useful as indicator of target label:
\nAnswer (From the Hosts):
\nTL;DR: This is not a data leak, we understand the reason this is happening and it is expected. Nothing to worry about.
\n\n\nand they are partly useful as indicator of target label
\n
Yes, this is expected: stdev is computed from the data, and very strong signals will deviate the value away from the \"noise-only\" value. This should already tell you how \"strong\" signals in the test set are, by the way :) .
\n\n\nAround 80% of test data seems to be affected by this
\n
This is also well understood. We wanted to give as much real data as possible in this challenge, but we didn't have a reliable way of generating more than what the LIGO detectors produced during the run, and we did not want to run into issues about having too much correlation among different samples. For this reason, we decided to generate synthetic data using Gaussian noise, for which a PSD (or ASD, or sqrtSX, they are all essentially equivalent in this context) needs to be specified.
\n\n\nIt would us sidechannel which is not present in real GW search data.
\n
This is not a problem in CW searches. For \"clean\" (relatively quiet) bands we are able to estimate the PSD of the noise with quite good accuracy, and that is sufficient for us to actually perform our analysis.
\nThe dataset description for the competition states:
\n\"Please note the presence of a small number of files labeled -1.\nPhysicists are currently unable to determine the status of these files.\"\nAs it turns out, these are easter eggs!.
\nIf you plot their signal magnitudes as 2D arrays, you'll see:
50f09e37e
\nA cartoon of two black holes about to collide.
62b0dd011
\nImage that was broadcast into the universe in 1974, as part of the SETI program.
\n
b7666b451
\nThe 2017 physics Nobel Prize winners (for the discovery of gravitational waves): Rainer Weiss, Barry Barish and Kip Thorne.
\n
An interesting discussion about methods for reducing the difference between the LB and the CV score.
\nThis competition is very different from the last one. The last contest was the signal of two black holes or neutron stars merging, which is very strong, and implementing a CNN would allow for some level of fit. But continuous gravitational waves are very weak, and even after scientists in the field have implemented various search methods and deep learning tools they have not found any real signal. If you read some of the papers you will see that even the most advanced NNs that have been tried have not reached traditional methods of search in terms of accuracy, and are usually very unsatisfactory after a SNR of less than 10.\n\nI think the huge gap between CV and LB is because 1) the training set is so small with only 600 data, there is so little variation in between that it may be difficult for the network to learn convolution methods in the face of low SNR samples, and 2) I haven't carefully checked the test set, but I think the composition of the test set must be very demanding and there must be a large number of low SNR samples.\n\nIn order to be able to bring CV and LB closer together, I think the most effective way to do this would be to construct a training set that has a large number of samples with a wide range of SNR values, rather than using 600 training samples and then trying various ML tricks.\nAn interesting and informative discussion about the posibility of additional instrumental artifacts (persistent line-like features) to the noise alongside the simulated signals in testing dataset.
\nYes, the artifacts like persistent lines do exist alongside with the CW signal and background noise. I've done some statistical testing of the whole dataset (test+train) and found them. \n\n\nAnother discussion about creating a robust CV strategy.
\nImportant Note by Chenglu
\nWith the generated dataset, I can reduce the gap to 0.1, CV ~0.77 and LB ~0.68, I think this competition is a matter of shrinking the gap, more precisely, it's generating a dataset that matches the distribution of test set.\nThank you for sharing this!!
\nChenglu is sharing with us a common bug that caused some problems (and you need to also be aware of it):
The bug: Normalizing the data this way:
\nnorm_data = data - data.() / (data.() - data.())\nThis is easily influenced by the value of max and min.
\nThe fix:
norm_data = data * 1e22\nThe fixed classifier only achieve 0.66 on classifying generating v.s. testing/training dataset.
\nThe post lists the differences between the two competitions:
\nThe raw data last year are 1d waves, whose x-axis is time, and y-axis is the amplitude of each timestamp. With CQT/CWT, the 1d wave can be converted into a 2d image(spectrum), which x-axis is time, y-axis is frequency and the intensity per unit pixel is the amplitude of a specific frequency within a specific timestamp.
The raw data are 2d images(spectrum), which x-axis is time, y-axis is frequency and the intensity per unit pixel is the amplitude of a specific frequency within a specific timestamp. (Which is the same as the data after CQT last year in format? I am not sure…)
SNR
\nThe SNR in the competition this year is much lower, which means that the signal this year is much weaker, resulting in the auc being much lower.
Important Addition by Alex Z: Physics-wise, last year competition dealt with the waves from compact binary objects, like close orbiting black holes or white dwarfs. This year we face fast spinning asymmetric neutron stars. Last year set-up provides much more explosive waveforms, and much stronger. Continuous GW we are hunting here were never detected so far, so we have a chance to help a real scientific discovery.
\n\nTL;DR: Vladimir Dropped a BOMB.
\n
It might be possible to reverse engineer the injected CW signal by using raw Ligo&Virgo data.
\nIn Short
\n\n\nWow.. well done on this!
\n
Igor Litvin proposed a method with some potential for improving your NN training.
He figered out that aproximately 20% of the gravitation wave spectra are distorted.
And think the distortion was made artificially because of it is exatly the same distortion for H and L shoulders of the interferometer.
It is possible that they multiplied by exactly the same function to H and exactly the same (but different from H) to L shoulder.
Great topic by Martin Kovacevic Buvinic about important insights about the data.
CV: The test data and train data are different: cv gap between validation and test is always large (~0.10). Also low correlation. - BUT! somehow a 20 folds cv model turned out to be much better.
Guess: Using 20 folds is only 5% of the data as validation for each fold, meaning we are only validating with 30 observations and this lead to overfitting, therefore this validation is not good, increasing the number of folds will just lead to overfitting.
Blending: Blending in most of the casses helps.
Data Generation: Adding more data usually improves the generalization of the model, if we check lb it actually improves so adding more data is helpful. The data simulated is the training data, and the best guess is that the key of the competition is to generate data that is similar to the test data.
The Main Insight: Using training data as validation is not optimal, because it is small and different compared to the test set. A good option would be to generate our own data that is similar to the test set and do traininig and validation with that data, also add the training data.
Vladimir Slaykovskiy asks about the easies way to combine generated data into the a single spectrogram.
\nsample = 0.9 * noise + 0.1 + signalAnswer (By The Hosts):
\nIf noise signal are Fourier amplitudes (i.e. the numbers you get get_stft_as_arrays) the generated SFTs have the same timestamps frequency, then yes.\n\nNote that you can literally do whatever you want to that data. For example, you wanted to create an instrumental artifact a circular shape, you could create such a shape a numpy array the appropriate array shape directly added to your data.\n\n`For example it correct to just weight-average them?`\n\nThe standard way which we do this by generating signals the amplitude of which already a certain fraction of your noiseF0: This may be tricky. The generated noise will be centered around this frequency. If a signal is created, it will be also a signal with this frequency (at the source; when it arrives to the detector, doppler and other effects will shift the frequency), so it will appear more or less on the center of the data. Range used on the datasets: [50, 400] Hz.
\nThere is a comment regarding the slicing in a way that F0 is not centered:
\nfreqs, times, sft_data = pyfstat.utils.get_sft_as_arrays(signal_writer.sftfilepath)
\nfirst_index = np.argmin(np.abs(freqs - 150.))
\nlast_index = np.argmin(np.abs(freqs - 150.2))
freqs = freqs[first_index:last_index+1]
\namplitudes = {key: val[first_index:last_index + 1, :]
\n for key, val in sft_data.items()}
print(\"*\" * 20)
\nprint(\"*\" * 20)
\nprint(f\"These should be 150. (got {freqs[0]}) \"
\n f\"and 150.2 (got {freqs[-1]}).\")
F1: This is technically the first derivative of the frequency (how it changs over time). Physically, the sources (neutron stars) lose energy over time because they emit the GWs, so this should be a negative number. A reasonable range could be [-1e-9, 0] Hz but that's up to us.
\nThere is also some interest for positive values.
\nh0: The amplitude of the signal. In the tutorials, the approach is to define it as a ratio of the sqrtSX. What makes the competition (and the real-world problem of finding continuous GWs) is that expected h0 will be between 1 and 2 orders of magnitude smaller than the amplitude of the noise. In my opinion, part of the challenge is to create your own signals with a distribution of h0 similar to the one used for the test set.
\nit should be mentioned SNR generation notebook and comments regarding cosi and value of h0
\nhttps://www.kaggle.com/code/chris62/generating-low-snr-gravity-waves
\nhttps://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/370202
\nRodrigo Tenorio
\nCOMPETITION HOST
\nPosted 15 days ago
\nThis post earned a bronze medal
arrow_drop_down
\nmore_vert
\nHi @forcewithme ,
You can have a look at equation 95 of this document where the specific dependency of SNR and cosi is discussed (in this case cosi is denoted as eta).
\nIn a nutshell, cosi=+-1 gives you a stronger signal, while cosi=0 gives you a weaker signal. The specific factor is cosi^4 + 6 * cosi^2 + 1 and that essentially means around a factor 8 of difference between both extreme cases.
", "rawMarkdown": "##### F0\nF0: This may be tricky. The generated noise will be centered around this frequency. If a signal is created, it will be also a signal with this frequency (at the source; when it arrives to the detector, doppler and other effects will shift the frequency), so it will appear more or less on the center of the data. Range used on the datasets: [50, 400] Hz.\n\nThere is a comment regarding the slicing in a way that F0 is not centered:\n\n##### Slice out the band of interest\nfreqs, times, sft_data = pyfstat.utils.get_sft_as_arrays(signal_writer.sftfilepath)\n\nfirst_index = np.argmin(np.abs(freqs - 150.))\nlast_index = np.argmin(np.abs(freqs - 150.2))\n\nfreqs = freqs[first_index:last_index+1]\namplitudes = {key: val[first_index:last_index + 1, :]\n for key, val in sft_data.items()}\n\nprint(\"*\" * 20)\nprint(\"*\" * 20)\nprint(f\"These should be 150. (got {freqs[0]}) \"\n f\"and 150.2 (got {freqs[-1]}).\")\n##### F1\nF1: This is technically the first derivative of the frequency (how it changs over time). Physically, the sources (neutron stars) lose energy over time because they emit the GWs, so this should be a negative number. A reasonable range could be [-1e-9, 0] Hz but that's up to us.\n\nThere is also some interest for positive values. \n\n##### h0 and cosi\nh0: The amplitude of the signal. In the tutorials, the approach is to define it as a ratio of the sqrtSX. What makes the competition (and the real-world problem of finding continuous GWs) is that expected h0 will be between 1 and 2 orders of magnitude smaller than the amplitude of the noise. In my opinion, part of the challenge is to create your own signals with a distribution of h0 similar to the one used for the test set.\n\nit should be mentioned SNR generation notebook and comments regarding cosi and value of h0\n\nhttps://www.kaggle.com/code/chris62/generating-low-snr-gravity-waves\n\nhttps://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/370202\n\nRodrigo Tenorio\nCOMPETITION HOST\nPosted 15 days ago\nThis post earned a bronze medal\n\narrow_drop_down\nmore_vert\nHi @forcewithme ,\n\nYou can have a look at equation 95 of this document where the specific dependency of SNR and cosi is discussed (in this case cosi is denoted as eta).\n\nIn a nutshell, cosi=+-1 gives you a stronger signal, while cosi=0 gives you a weaker signal. The specific factor is cosi^4 + 6 * cosi^2 + 1 and that essentially means around a factor 8 of difference between both extreme cases.", "votes": 1 }, { "id": 2075122, "postDate": "2022-12-25T04:58:08.017Z", "content": "well written.
", "rawMarkdown": "well written." }, { "id": 2075017, "postDate": "2022-12-24T23:23:35.647Z", "content": "Wow! That's a lot of precious stuff. I hope they could read it.
\nAt least, for those that didn't read on the last ending 10 days, it'll remain on the Cloud for the next Gravitational Waves competitions.
\nWonderful job The Devastator.
", "rawMarkdown": "Wow! That's a lot of precious stuff. I hope they could read it. \n\nAt least, for those that didn't read on the last ending 10 days, it'll remain on the Cloud for the next Gravitational Waves competitions.\n\nWonderful job The Devastator." }, { "id": 2078389, "postDate": "2022-12-28T09:45:35.563Z", "content": "This is super helpful, thanks!
", "rawMarkdown": "This is super helpful, thanks!" }, { "id": 2077379, "postDate": "2022-12-27T14:21:31.120Z", "content": "Thanks for your summary.
", "rawMarkdown": "Thanks for your summary." }, { "id": 2076873, "postDate": "2022-12-27T00:44:32.893Z", "content": "Thanks for the helpful article.
", "rawMarkdown": "Thanks for the helpful article. " } ], "comments": [ { "id": 2075382, "author_name": "George Chirita", "author_url": "", "post_date": "2022-12-25T10:27:03.427000", "content": "F0: This may be tricky. The generated noise will be centered around this frequency. If a signal is created, it will be also a signal with this frequency (at the source; when it arrives to the detector, doppler and other effects will shift the frequency), so it will appear more or less on the center of the data. Range used on the datasets: [50, 400] Hz.
\nThere is a comment regarding the slicing in a way that F0 is not centered:
\nfreqs, times, sft_data = pyfstat.utils.get_sft_as_arrays(signal_writer.sftfilepath)
\nfirst_index = np.argmin(np.abs(freqs - 150.))
\nlast_index = np.argmin(np.abs(freqs - 150.2))
freqs = freqs[first_index:last_index+1]
\namplitudes = {key: val[first_index:last_index + 1, :]
\n for key, val in sft_data.items()}
print(\"*\" * 20)
\nprint(\"*\" * 20)
\nprint(f\"These should be 150. (got {freqs[0]}) \"
\n f\"and 150.2 (got {freqs[-1]}).\")
F1: This is technically the first derivative of the frequency (how it changs over time). Physically, the sources (neutron stars) lose energy over time because they emit the GWs, so this should be a negative number. A reasonable range could be [-1e-9, 0] Hz but that's up to us.
\nThere is also some interest for positive values.
\nh0: The amplitude of the signal. In the tutorials, the approach is to define it as a ratio of the sqrtSX. What makes the competition (and the real-world problem of finding continuous GWs) is that expected h0 will be between 1 and 2 orders of magnitude smaller than the amplitude of the noise. In my opinion, part of the challenge is to create your own signals with a distribution of h0 similar to the one used for the test set.
\nit should be mentioned SNR generation notebook and comments regarding cosi and value of h0
\nhttps://www.kaggle.com/code/chris62/generating-low-snr-gravity-waves
\nhttps://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/370202
\nRodrigo Tenorio
\nCOMPETITION HOST
\nPosted 15 days ago
\nThis post earned a bronze medal
arrow_drop_down
\nmore_vert
\nHi @forcewithme ,
You can have a look at equation 95 of this document where the specific dependency of SNR and cosi is discussed (in this case cosi is denoted as eta).
\nIn a nutshell, cosi=+-1 gives you a stronger signal, while cosi=0 gives you a weaker signal. The specific factor is cosi^4 + 6 * cosi^2 + 1 and that essentially means around a factor 8 of difference between both extreme cases.
", "votes": 1, "replies": [] }, { "id": 2075122, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-12-25T04:58:08.017000", "content": "well written.
", "votes": 0, "replies": [] }, { "id": 2075017, "author_name": "Marília Prata", "author_url": "", "post_date": "2022-12-24T23:23:35.647000", "content": "Wow! That's a lot of precious stuff. I hope they could read it.
\nAt least, for those that didn't read on the last ending 10 days, it'll remain on the Cloud for the next Gravitational Waves competitions.
\nWonderful job The Devastator.
", "votes": 0, "replies": [] }, { "id": 2078389, "author_name": "tainangao", "author_url": "", "post_date": "2022-12-28T09:45:35.563000", "content": "This is super helpful, thanks!
", "votes": 0, "replies": [] }, { "id": 2077379, "author_name": "Ursula Lu", "author_url": "", "post_date": "2022-12-27T14:21:31.120000", "content": "Thanks for your summary.
", "votes": 0, "replies": [] }, { "id": 2076873, "author_name": "plasmabio", "author_url": "", "post_date": "2022-12-27T00:44:32.893000", "content": "Thanks for the helpful article.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2074716": "### What do we know so far? \n##### 10 Days to go!\n_____\nSave yourself some time and get caught up on the latest updates and important discussions that have taken place prior to the competition deadline.\nThe competition ends in 10 days and there is still time to improve your solution. Here are some of the important topics that have been discussed in the Kaggle forums.\n_____\n\n\n##### Background\n\n- The data is H-U-G-E and comes in hdf5 format, [chazzer](https://www.kaggle.com/chazzer) had written a [notebook on how to read the hdf5 files](https://www.kaggle.com/code/chazzer/how-to-read-the-hdf5-files/)\n\n- Shortly after [JohnM](https://www.kaggle.com/jpmiller) pushed this one step further and created [HDF files with indexed dataframes for the training set](https://www.kaggle.com/datasets/jpmiller/simplified-dataset). The index on each frame is the frequency array from the original file, and the columns are the timestamps.\n\n**Reading Example**\n\n```python\ndf_h = pd.read_hdf(\"../input/simplified-dataset/001121a05.h5\", key = 'h')\n```\n\n- Great [list](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/357682) of learning resources had been published by [Laura Fink](https://www.kaggle.com/allunia)\n - [Gravitational Wave Open Science Center - Tutorials](https://www.gw-openscience.org/tutorials/)\n - [GWPy](https://gwpy.github.io/docs/stable/index.html)\n - [Video about the Michelson Interferometer](https://www.youtube.com/watch?v=tQ_teIUb3tE)\n - [Video about gravitational waves](https://www.youtube.com/watch?v=FlDtXIBrAYE&t=1s)\n - [Great tutorial video about Fourier transforms](https://www.youtube.com/watch?v=spUNpyF58BY&t=1s)\n - [Video about Denoising data with Fast Fourier Transforms](https://www.youtube.com/watch?v=s2K1JfNR7Sc)\n \n_____\n\n##### Best CV-LB Results ([Mark Baushenko](https://www.kaggle.com/e0xextazy))\n\n- **Important** Comments by [DrHB](drhabib)\n - \"Without `augs` the gap (CV-LB) is bigger\"\n - \"After running some experiments .. I think the biggest challenge of this competition will be to build a proper validation set\" \n \n_____\n\n\n##### Understanding the Data ([Ravi Shah](https://www.kaggle.com/ravishah1))\n\nAn [amazing intro post](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/358444) by [Ravi Shah](https://www.kaggle.com/ravishah1) going through all important details about the data of this competition. \n\n**In short (check the original for more info)**\n\n\n\n- L1 - data from the LIGO Livingston interferometer\n- H1 - data from the LIGO Hanford interferometer\n- STFs - Short-time Fourier Transforms (SFTs) - shape (360, n)\n- timestamps - the timestamps that the STFs correspond to - shape (n,)\n- frequency Hz - the range frequencies measured by the detectors - shape (360,)\n\n**Competition Goal**\n\nThe goal of this competition is to create a model that can detect continuous gravitational-wave signals. These are weak yet long-lasting signals emitted by rapidly-spinning neutron stars within noisy data. \n\n> - **Target is 1 if signal is present else 0**\n\n\n**LIGO** (where the data comes from) is a detector in Livingston, Louisiana and Hanford, Washington. The detector has a L shape with equal-length arms connected to a corner station. When a gravitational wave passes by, one leg of the detector is shortened while the other is lengthened. \n\n**The interference makes a shift that we can analyze.**\n\n\n**We Get A Continuous Gravitational Waves**\n\nA continuous gravitational-wave signal from a Galactic neutron star will look almost perfectly constant in both frequency and amplitude.\n\n\n\n**But over long durations: The frequency of the signal slowly changes, because:**\n\n- As the neutron star emits gravitational and electromagnetic waves, it loses energy which causes it to rotate more slowly\n- The detector here on Earth is moving with respect to the neutron star. This changes the frequency of the gravitational waves observed in the detector.\n\n**Short-time Fourier Transforms (SFTs)**\n\n- Short-time Fourier Transforms (SFTs) can be used as a way of quantifying the change of a nonstationary signal’s frequency and phase content over time.\n- Very common type of signal preprocessing technique.\n- This what we are given to work with (hdf5 files).\n\n\n\n\n_____\n\n\n##### Timestamps and Frequencies Insights ([Mark Wijkhuizen](https://www.kaggle.com/markwijkhuizen))\n\n**There seems to be 2 differences of our data in comparison to conventional SFT's**\n\n- **The time delta between each timestamp is inconsistent:** Time between measurement in the SFT's can vary greatly.\n- The frequency range is consistently ~0.2Hz, **BUT the min/max frequency ranges from around ~50to ~498Hz.**\n\n- **We should not interpret all spectrograms as continuous sounds:** When taking a look at the timestamps they start at 2009-03-27/28 and end on 2009-07-25/26/27. The recordings are thus gathered during 4 months with often 30 minutes between the recordings. \n\n_____\n\n\n##### Minimum float32 is 1e-38 and data**2 is 1e-44 ([🐢 Jun Koda](https://www.kaggle.com/junkoda))\n\nImportant [post](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/361312) by [🐢 Jun Koda](https://www.kaggle.com/junkoda) sharing with us that since the smallest positive number float32 (~ 1e-38) but on the data it is ~1e-22. \n\n**Once we square the data, they go beyond this limit.**\n\n> np.float32(1.23456e-44) => 1.3e-44.\n\n- float32 has about 8 significant digits and 1.23456e-44 will be expressed as 0.0000012e-38 with float32, only 2 digits remaining.\n- Suggestion: that we multiply the data by 1e21 or 1e22.\n\n_____\n\n\n##### PyFstat Signal Parameters ([Mark Wijkhuizen](https://www.kaggle.com/markwijkhuizen))\n\n> Another great [post](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/361562) by ([Mark Wijkhuizen](https://www.kaggle.com/markwijkhuizen))\n\nPyFstat has many parameters but it's documentation however is not very helpful for explaining them.\n\n\n**[Victor Gonzalez](https://www.kaggle.com/darkbitur)** Explained with an incredible comment everything about all the paramaeters. \n\n- You can use a helper function to inject some parameters based on priors:\n \n```python\n{**pyfstat.injection_parameters.isotropic_amplitude_priors}\n{'cosi': {'uniform': {'low': -1.0, 'high': 1.0}}, 'psi': {'uniform': {'low': -0.7853981633974483, 'high': 0.7853981633974483}}, 'phi': {'uniform': {'low': 0, 'high': 6.283185307179586}}}\n```\n\n**More Parameters**\n\n- **cosi:** Cosine of the angle between the source and us. Range: [-1, 1]\n- **psi and phi:** polarization angle and phase. Ranges: [-pi/2, pi/2] and [0, 2*pi]\n- **Alpha:** Right ascension of the source's position on the sky. Range: [0, 2*pi]\n- **Delta:** Declination of the source's position on the sky. Range: [-pi/2, pi/2]\n- **Band:** This is just the width of the frequency \"slice\" to be taken. It's 0.2 hz for all files in the train and test datasets, which translates to 360 \"frequency lines\".\n- **F0:** This may be tricky. The generated noise will be centered around this frequency. If a signal is created, it will be also a signal with this frequency (at the source; when it arrives to the detector, doppler and other effects will shift the frequency), so it will appear more or less on the center of the data. Range used on the datasets: [50, 400] Hz.\n- **F1:** This is technically the first derivative of the frequency (how it changs over time). Physically, the sources (neutron stars) lose energy over time because they emit the GWs, so this should be a negative number. A reasonable range could be [-1e-9, 0] Hz but that's up to us.\n- **F2:** This is guess it's the second derivative, which maybe is 0 for the purposes of this competition.\n- **h0:** The amplitude of the signal. In the tutorials, the approach is to define it as a ratio of the sqrtSX. What makes the competition (and the real-world problem of finding continuous GWs) is that expected h0 will be between 1 and 2 orders of magnitude smaller than the amplitude of the noise. In my opinion, part of the challenge is to create your own signals with a distribution of h0 similar to the one used for the test set.\n- **psi** is actually defined within [-pi /4 , pi / 4] (and it's periodic, so anything beyond that folds back onto this range).\n- **F2** is another term describing the intrinsic evolution of a gravitational wave. As @darkbitur correctly guesses, it's set to 0 in this competition as we only focus on frequency and spindown.\n- **tp, asini, period** are parameters used to describe neutron stars in binary systems, and they affect the frequency evolution of the signal. You can safely ignore those in this competition (but do feel free to play around with them if you wish to!).\n\n_____\n\n\n##### Can we assume the same signal trajectories between Hanford & Livingston? ([IkaroSilva](https://www.kaggle.com/ikarosilva))\n\nQuestion [asked](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/361655) by [IkaroSilva](https://www.kaggle.com/ikarosilva)\n\n- Can we assume the same signal trajectories between Hanford & Livingston?\n\n**Answer (By The Hosts):**\n\n- The frequency of a signal as seen by a detector a certain time t (what you call it S(t) in your post) does generally depend on the detector, the reason being that different detectors move \"differently\" around the Sun as they are in slightly different positions on Earth.\n\n- This distinction may not be that noticeable for some cases if we use the two Advanced LIGO detectors, since they are relatively close to each other, but becomes quite apparent once you include other detectors such as Virgo, which is located in Italy.\n\n_____\n\n\n##### What is your batch size and LB? ([dragon zhang](https://www.kaggle.com/dragonzhang))\n\nA [good] question about the relation beween the batch-size and generalization performance by [dragon zhang](https://www.kaggle.com/dragonzhang).\n\n\n**Short Answer:**\n\n```\nFriends dont let friends use minibatches larger than 32.\n[Yann LeCun](https://twitter.com/ylecun/status/989610208497360896).\n```\n\n_____\n\n\n##### Recap of the Top Solutions from the Previous G2Net Competition ([Sinan Calisir](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363280))\n\n\nAn amazing solutions [summary](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363280) by [Sinan Calisir](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363280) of all winning solutions of the previous competition.\n\n\n_____\n\n\n##### Weird property of std(spectrogram) in train data ([Josef Slavicek\n](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363460))\n\n\n**Wierd Observation:** When computing std of spectrogram in complex64, I'm getting almost allways the same value of 1.4973568266036865e-22. If I do the same computation in complex128, the values starts do differ - and they are partly useful as indicator of target label:\n\n\n**Answer (From the Hosts):**\n\n**TL;DR:** This is not a data leak, we understand the reason this is happening and it is expected. Nothing to worry about.\n\n> and they are partly useful as indicator of target label\n\nYes, this is expected: stdev is computed from the data, and very strong signals will deviate the value away from the \"noise-only\" value. This should already tell you how \"strong\" signals in the test set are, by the way :) .\n\n> Around 80% of test data seems to be affected by this\n\nThis is also well understood. We wanted to give as much real data as possible in this challenge, but we didn't have a reliable way of generating more than what the LIGO detectors produced during the run, and we did not want to run into issues about having too much correlation among different samples. For this reason, we decided to generate synthetic data using Gaussian noise, for which a PSD (or ASD, or sqrtSX, they are all essentially equivalent in this context) needs to be specified.\n\n> It would us sidechannel which is not present in real GW search data.\n\nThis is not a problem in CW searches. For \"clean\" (relatively quiet) bands we are able to estimate the PSD of the noise with quite good accuracy, and that is sufficient for us to actually perform our analysis. \n\n_____\n\n##### Regarding Those -1 Labels ([PaulG](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363734))\n\n\n- [PaulG](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363734) **Found an easter egg** left by the competition Hosts! \n\nThe dataset description for the competition states:\n``` \n\"Please note the presence of a small number of files labeled -1.\nPhysicists are currently unable to determine the status of these files.\"\n```\n\nAs it turns out, these are easter eggs!.\nIf you plot their signal magnitudes as 2D arrays, you'll see:\n\n\n**50f09e37e**\nA cartoon of two black holes about to collide.\n\n\n\n**62b0dd011**\nImage that was broadcast into the universe in 1974, as part of the SETI program.\n\n\n**b7666b451**\nThe 2017 physics Nobel Prize winners (for the discovery of gravitational waves): Rainer Weiss, Barry Barish and Kip Thorne.\n\n\n_____\n\n\n\n##### High validation accuracy, low LB score ([Simone Bavera](https://www.kaggle.com/simonebvr))\n\nAn [interesting](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363810) discussion about methods for reducing the difference between the LB and the CV score. \n\n\n- **Important** Answer by [Chen Lin](https://www.kaggle.com/chenlin1999)\n\n\n```\nThis competition is very different from the last one. The last contest was the signal of two black holes or neutron stars merging, which is very strong, and implementing a CNN would allow for some level of fit. But continuous gravitational waves are very weak, and even after scientists in the field have implemented various search methods and deep learning tools they have not found any real signal. If you read some of the papers you will see that even the most advanced NNs that have been tried have not reached traditional methods of search in terms of accuracy, and are usually very unsatisfactory after a SNR of less than 10.\n\nI think the huge gap between CV and LB is because 1) the training set is so small with only 600 data, there is so little variation in between that it may be difficult for the network to learn convolution methods in the face of low SNR samples, and 2) I haven't carefully checked the test set, but I think the composition of the test set must be very demanding and there must be a large number of low SNR samples.\n\nIn order to be able to bring CV and LB closer together, I think the most effective way to do this would be to construct a training set that has a large number of samples with a wide range of SNR values, rather than using 600 training samples and then trying various ML tricks.\n```\n\n\n_____\n\n\n\n##### Possible instrumental artifacts in training/testing dataset ([Alex Z](https://www.kaggle.com/alexz0))\n\n\nAn [interesting](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/364854) and informative discussion about the posibility of additional instrumental artifacts (persistent line-like features) to the noise alongside the simulated signals in testing dataset.\n\n\n- **Important** Answer by [Konstantin Dmitriev](https://www.kaggle.com/kdmitrie):\n\n```\nYes, the artifacts like persistent lines do exist alongside with the CW signal and background noise. I've done some statistical testing of the whole dataset (test+train) and found them. \n\n\n\n```\n\n_____\n\n\n\n##### Gap of CV and LB ([Chenglu](https://www.kaggle.com/snaker))\n\n\nAnother [discussion](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/364169) about creating a robust CV strategy.\n\n\n**Important Note** by [Chenglu](https://www.kaggle.com/snaker)\n\n```\nWith the generated dataset, I can reduce the gap to 0.1, CV ~0.77 and LB ~0.68, I think this competition is a matter of shrinking the gap, more precisely, it's generating a dataset that matches the distribution of test set.\n```\n\n_____\n\n\n##### Generating samples that match the train / test distribution ([Chenglu](https://www.kaggle.com/snaker))\n\n\nThank you for sharing this!!\n[Chenglu](https://www.kaggle.com/snaker) is sharing with us a common bug that caused some problems (and you need to also be aware of it):\n\nThe bug: Normalizing the data this way:\n\n```python\nnorm_data = data - data.min() / (data.max() - data.min())\n```\n\nThis is easily influenced by the value of max and min.\nThe fix:\n```\nnorm_data = data * 1e22\n```\n\nThe fixed classifier only achieve 0.66 on classifying generating v.s. testing/training dataset.\n\n_____\n\n\n\n##### What is the difference between the G2Net this year and last year? ([ForcewithMe](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/366451))\n\n**The post lists the differences between the two competitions:**\n\n- The raw data last year are 1d waves, whose x-axis is time, and y-axis is the amplitude of each timestamp. With CQT/CWT, the 1d wave can be converted into a 2d image(spectrum), which x-axis is time, y-axis is frequency and the intensity per unit pixel is the amplitude of a specific frequency within a specific timestamp.\n\n- The raw data are 2d images(spectrum), which x-axis is time, y-axis is frequency and the intensity per unit pixel is the amplitude of a specific frequency within a specific timestamp. (Which is the same as the data after CQT last year in format? I am not sure…)\n\n**SNR**\n- The SNR in the competition this year is much lower, which means that the signal this year is much weaker, resulting in the auc being much lower.\n\n- **Important** Addition by [Alex Z](https://www.kaggle.com/alexz0): Physics-wise, last year competition dealt with the waves from compact binary objects, like close orbiting black holes or white dwarfs. This year we face fast spinning asymmetric neutron stars. Last year set-up provides much more explosive waveforms, and much stronger. Continuous GW we are hunting here were never detected so far, so we have a chance to help a real scientific discovery.\n\n_____\n\n\n\n##### Reverse engineering 20% of test samples with external data. ([Vladimir Slaykovskiy](https://www.kaggle.com/vslaykovsky))\n\n> **TL;DR:** Vladimir Dropped a BOMB. \n\n**It might be possible to reverse engineer the injected CW signal by using raw Ligo&Virgo data.**\n\n- [code](https://www.kaggle.com/vslaykovsky/g2net-winning-strategy-with-external-data)\n\n**In Short**\n\n- Get raw detector data at https://www.gw-openscience.org/archive/O3a_4KHZ_R1/\n- Generate SFTs using instructions at https://youtu.be/A9iWRcmG0Rs?t=2199 (you'll likely need to google for more details on this step). Use GPS timestamps from test set to produce SFTs.\n- Read generated SFTs using pyfstat.utils.sft.get_sft_as_arrays\n- Match generated SFTs with real samples from test set.\n- Find the difference between SFTs and real samples. difference > EPS -> label==1; difference <= EPS -> label==0.\n\n> Wow.. well done on this!\n\n_____\n\n\n\n##### Score improvement G wave detection 20% data ([Igor Litvin](https://www.kaggle.com/igorlitvin))\n\n- [Igor Litvin](https://www.kaggle.com/igorlitvin) proposed a method with some potential for improving your NN training.\n\n- He figered out that aproximately 20% of the gravitation wave spectra are distorted.\n- And think the distortion was made artificially **because of it is exatly the same distortion for H and L shoulders of the interferometer.**\n- It is possible that they multiplied by exactly the same function to H and exactly the same (but different from H) to L shoulder.\n\n- [Code here](https://www.kaggle.com/code/igorlitvin/l-and-h-distortion-of-g-wave-help)\n\n_____\n\n\n\n##### Some Insights About Validation Strategy ([Martin Kovacevic Buvinic](https://www.kaggle.com/ragnar123))\n\n- Great topic by [Martin Kovacevic Buvinic](https://www.kaggle.com/ragnar123) about important insights about the data.\n\n- **CV:** The test data and train data are different: cv gap between validation and test is always large (~0.10). Also low correlation. - BUT! somehow a 20 folds cv model turned out to be much better. \n- **Guess:** Using 20 folds is only 5% of the data as validation for each fold, meaning we are only validating with 30 observations and this lead to overfitting, therefore this validation is not good, increasing the number of folds will just lead to overfitting.\n- **Blending:** Blending in most of the casses helps.\n- **Data Generation:** Adding more data usually improves the generalization of the model, if we check lb it actually improves so adding more data is helpful. The data simulated is the training data, and the best guess is that the key of the competition is to generate data that is similar to the test data.\n- **The Main Insight:** Using training data as validation is not optimal, because it is small and different compared to the test set. A good option would be to generate our own data that is similar to the test set and do traininig and validation with that data, also add the training data.\n\n_____\n\n\n\n##### Merging generated signal into noise ([Vladimir Slaykovskiy](https://www.kaggle.com/vslaykovsky))\n\n[Vladimir Slaykovskiy](https://www.kaggle.com/vslaykovsky) asks about the easies way to combine generated data into the a single spectrogram.\n\n- For example is it correct to just weight-average them?\n`sample = 0.9 * noise + 0.1 + signal`\n\n**Answer (By The Hosts):**\n\n\n```python\nIf noise and signal are Fourier amplitudes (i.e. the complex numbers you get from get_stft_as_arrays) and the generated SFTs have the same timestamps and frequency, then yes.\n\nNote that you can literally do whatever you want to that data. For example, if you wanted to create an instrumental artifact with a circular shape, you could create such a shape in a numpy array with the appropriate array shape and directly added to your data.\n\n`For example is it correct to just weight-average them?`\n\nThe standard way in which we do this is by generating signals the amplitude of which is already a certain fraction of your noise's amplitude (so we can just add both arrays).\n```\n\n\n\n\n\n\n", "2075382": "##### F0\nF0: This may be tricky. The generated noise will be centered around this frequency. If a signal is created, it will be also a signal with this frequency (at the source; when it arrives to the detector, doppler and other effects will shift the frequency), so it will appear more or less on the center of the data. Range used on the datasets: [50, 400] Hz.\n\nThere is a comment regarding the slicing in a way that F0 is not centered:\n\n##### Slice out the band of interest\nfreqs, times, sft_data = pyfstat.utils.get_sft_as_arrays(signal_writer.sftfilepath)\n\nfirst_index = np.argmin(np.abs(freqs - 150.))\nlast_index = np.argmin(np.abs(freqs - 150.2))\n\nfreqs = freqs[first_index:last_index+1]\namplitudes = {key: val[first_index:last_index + 1, :]\n for key, val in sft_data.items()}\n\nprint(\"*\" * 20)\nprint(\"*\" * 20)\nprint(f\"These should be 150. (got {freqs[0]}) \"\n f\"and 150.2 (got {freqs[-1]}).\")\n##### F1\nF1: This is technically the first derivative of the frequency (how it changs over time). Physically, the sources (neutron stars) lose energy over time because they emit the GWs, so this should be a negative number. A reasonable range could be [-1e-9, 0] Hz but that's up to us.\n\nThere is also some interest for positive values. \n\n##### h0 and cosi\nh0: The amplitude of the signal. In the tutorials, the approach is to define it as a ratio of the sqrtSX. What makes the competition (and the real-world problem of finding continuous GWs) is that expected h0 will be between 1 and 2 orders of magnitude smaller than the amplitude of the noise. In my opinion, part of the challenge is to create your own signals with a distribution of h0 similar to the one used for the test set.\n\nit should be mentioned SNR generation notebook and comments regarding cosi and value of h0\n\nhttps://www.kaggle.com/code/chris62/generating-low-snr-gravity-waves\n\nhttps://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/370202\n\nRodrigo Tenorio\nCOMPETITION HOST\nPosted 15 days ago\nThis post earned a bronze medal\n\narrow_drop_down\nmore_vert\nHi @forcewithme ,\n\nYou can have a look at equation 95 of this document where the specific dependency of SNR and cosi is discussed (in this case cosi is denoted as eta).\n\nIn a nutshell, cosi=+-1 gives you a stronger signal, while cosi=0 gives you a weaker signal. The specific factor is cosi^4 + 6 * cosi^2 + 1 and that essentially means around a factor 8 of difference between both extreme cases.", "2075122": "well written.", "2075017": "Wow! That's a lot of precious stuff. I hope they could read it. \n\nAt least, for those that didn't read on the last ending 10 days, it'll remain on the Cloud for the next Gravitational Waves competitions.\n\nWonderful job The Devastator.", "2078389": "This is super helpful, thanks!", "2077379": "Thanks for your summary.", "2076873": "Thanks for the helpful article. " } }