\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F59264%2Fe913756c71cfdb5b926f70543095733c%2F__results___6_1.png?generation=1667324396733833&alt=media\")
Hello @rodrigotenorio , @michaeljwill , I discovered very surprising property of train data. If I compute std of spectrogram in complex64, I'm getting almost allways the same value of 1.4973568266036865e-22. If I do the same computation in complex128, the values starts do differ - and they are partly useful as indicator of target label:
\n
The image is taken from notebook which demonstrate the behavior.
\nMy guess is, that most of training data was generated with single value of sqrtSX. Is this property of data known and considered as OK? For me, it feels almost like target label leak :-)
", "messages": [ { "id": 2013187, "postDate": "2022-11-01T17:45:55.097Z", "content": "Hello @rodrigotenorio , @michaeljwill , I discovered very surprising property of train data. If I compute std of spectrogram in complex64, I'm getting almost allways the same value of 1.4973568266036865e-22. If I do the same computation in complex128, the values starts do differ - and they are partly useful as indicator of target label:
\n
The image is taken from notebook which demonstrate the behavior.
\nMy guess is, that most of training data was generated with single value of sqrtSX. Is this property of data known and considered as OK? For me, it feels almost like target label leak :-)
", "rawMarkdown": "Hello @rodrigotenorio , @michaeljwill , I discovered very surprising property of train data. If I compute std of spectrogram in complex64, I'm getting almost allways the same value of 1.4973568266036865e-22. If I do the same computation in complex128, the values starts do differ - and they are partly useful as indicator of target label:\n\n\nThe image is taken from [notebook](https://www.kaggle.com/code/josefslavicek/g2net-weird-behavior-of-std-spectrogram) which demonstrate the behavior.\n\nMy guess is, that most of training data was generated with single value of sqrtSX. Is this property of data known and considered as OK? For me, it feels almost like target label leak :-)", "votes": 11 }, { "id": 2014503, "postDate": "2022-11-02T15:59:36.067Z", "content": "Hi @josefslavicek , all, thank you very much for publicly starting this thread.
\nWe (@michaeljwill and myself) had a brief discussion about this, I hope the following clarifies what is going on.
\ntl;dr This is not a data leak, we understand the reason this is happening and it is expected. Nothing to worry about.
\n\n\nand they are partly useful as indicator of target label
\n
Yes, this is expected: stdev is computed from the data, and very strong signals will deviate the value away from the \"noise-only\" value. This should already tell you how \"strong\" signals in the test set are, by the way :) .
\n\n\nAround 80% of test data seems to be affected by this
\n
This is also well understood. We wanted to give as much real data as possible in this challenge, but we didn't have a reliable way of generating more than what the LIGO detectors produced during the run, and we did not want to run into issues about having too much correlation among different samples. For this reason, we decided to generate synthetic data using Gaussian noise, for which a PSD (or ASD, or sqrtSX, they are all essentially equivalent in this context) needs to be specified.
\n\nIt would us sidechannel which is not present in real GW search data.
\n
This is not a problem in CW searches. For \"clean\" (relatively quiet) bands we are able to estimate the PSD of the noise with quite good accuracy, and that is sufficient for us to actually perform our analysis. The relevant bit is that, regardless of that PSD, the relevant quantity when it comes to detecting a signal is the SNR (which is roughly a ratio between signal amplitude h0 and noise amplitude ASD). Signals are usually so weak (read \"SNR is low\") that we cannot use PSD as a trigger for signals; in fact, high PSDs are usually related to noise artifacts.
All in all, the test set portraits a data distribution which is a close as possible to a general CW search as we could generate. There is a significant fraction of simulated data, as we state in the data page, and that is due to the impossibility of properly and reliably generating \"real data\" on demand, but that does not drift this competition away from the sort of \"realistic\" setup we want you to focus on.
\nWe are happy to clarify any other concerns, so please keep raising any issues you may be concerned :)
\nCheers,
", "rawMarkdown": "Hi @josefslavicek , all, thank you very much for publicly starting this thread.\n\nWe (@michaeljwill and myself) had a brief discussion about this, I hope the following clarifies what is going on.\n\ntl;dr This is not a data leak, we understand the reason this is happening and it is expected. Nothing to worry about.\n\n> and they are partly useful as indicator of target label\n\nYes, this is expected: stdev is computed from the data, and very strong signals will deviate the value away from the \"noise-only\" value. This should already tell you how \"strong\" signals in the test set are, by the way :) .\n\n > Around 80% of test data seems to be affected by this\n\nThis is also well understood. We wanted to give as much real data as possible in this challenge, but we didn't have a reliable way of generating more than what the LIGO detectors produced during the run, and we did not want to run into issues about having too much correlation among different samples. For this reason, we decided to generate synthetic data using Gaussian noise, for which a PSD (or ASD, or `sqrtSX`, they are all essentially equivalent in this context) needs to be specified. \n\n> It would us sidechannel which is not present in real GW search data.\n\nThis is not a problem in CW searches. For \"clean\" (relatively quiet) bands we are able to estimate the PSD of the noise with quite good accuracy, and that is sufficient for us to actually perform our analysis. The relevant bit is that, regardless of that PSD, the relevant quantity when it comes to detecting a signal is the SNR (which is roughly a ratio between signal amplitude `h0` and noise amplitude `ASD`). Signals are usually so weak (read \"SNR is low\") that we cannot use PSD as a trigger for signals; in fact, high PSDs are usually related to noise artifacts.\n\nAll in all, the test set portraits a data distribution which is a close as possible to a general CW search as we could generate. There is a significant fraction of simulated data, as we state in the data page, and that is due to the impossibility of properly and reliably generating \"real data\" on demand, but that does not drift this competition away from the sort of \"realistic\" setup we want you to focus on.\n\nWe are happy to clarify any other concerns, so please keep raising any issues you may be concerned :)\n\nCheers,", "votes": 3, "replies": [ { "id": 2014681, "postDate": "2022-11-02T18:40:17.017Z", "content": "OK, thanks for clarification.
", "rawMarkdown": "OK, thanks for clarification." } ] }, { "id": 2013238, "postDate": "2022-11-01T18:29:38.083Z", "content": "\"My guess is, that most of training data was generated with single value of sqrtSX. \"
\nYep. 5e-24 or so.
FYI, I have evaluated using predictions from my NN model and found that most of the samples where the std in complex128 judged outlier are easily true positives.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4759954%2F13d4e2af0594b5c2920159ca68fae6c7%2F20221102.png?generation=1667398280554223&alt=media\")
Getting same std with complex64 is a property of float32
\nhttps://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/361312
Signal contribute to std (total power) and correlated with label, but I don't think it is a strong estimator, and no worry for a leak. Test data also contain real noise, which are far from the train sqrtSX.
", "rawMarkdown": "Getting same std with complex64 is a property of float32\nhttps://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/361312\n\nSignal contribute to std (total power) and correlated with label, but I don't think it is a strong estimator, and no worry for a leak. Test data also contain real noise, which are far from the train sqrtSX.", "votes": 1, "replies": [ { "id": 2013747, "postDate": "2022-11-02T05:38:54.063Z", "content": "\n\nSignal contribute to std (total power) and correlated with label
\n
Yes, but if you generate big part of your dataset with just single value of sqrtSX, you get something what IMO you should not have: very good prior knowledge of std(spectrogram) for negative datapoints. You know it up to 3 decimal places. If people around LIGO have similarly good knowledge of noise in their detector, then everything is OK. If they don't, then this competition will be still fair, but not as useful as it could be. It would give us sidechannel which is not present in real GW search data.
Around 80% of test data seems to be affected by this.
", "rawMarkdown": "> Signal contribute to std (total power) and correlated with label\n\nYes, but if you generate big part of your dataset with just single value of sqrtSX, you get something what IMO you **should not** have: very good prior knowledge of `std(spectrogram)` for negative datapoints. You know it up to 3 decimal places. If people around LIGO have similarly good knowledge of noise in their detector, then everything is OK. If they don't, then this competition will be still fair, but not as useful as it could be. It would give us sidechannel which is not present in real GW search data. \n\nAround 80% of test data seems to be affected by this.", "votes": 1 }, { "id": 2013992, "postDate": "2022-11-02T08:40:47.510Z", "content": "I see, I agree with that. We know the statistics of the simulated noise very well, and those 80% are easier than real data.
", "rawMarkdown": "I see, I agree with that. We know the statistics of the simulated noise very well, and those 80% are easier than real data." }, { "id": 2014040, "postDate": "2022-11-02T09:16:28.113Z", "content": "Don't pay too much attention to the train set. Test set is very different.
", "rawMarkdown": "Don't pay too much attention to the train set. Test set is very different." } ] } ], "comments": [ { "id": 2014503, "author_name": "Rodrigo Tenorio", "author_url": "", "post_date": "2022-11-02T15:59:36.067000", "content": "Hi @josefslavicek , all, thank you very much for publicly starting this thread.
\nWe (@michaeljwill and myself) had a brief discussion about this, I hope the following clarifies what is going on.
\ntl;dr This is not a data leak, we understand the reason this is happening and it is expected. Nothing to worry about.
\n\n\nand they are partly useful as indicator of target label
\n
Yes, this is expected: stdev is computed from the data, and very strong signals will deviate the value away from the \"noise-only\" value. This should already tell you how \"strong\" signals in the test set are, by the way :) .
\n\n\nAround 80% of test data seems to be affected by this
\n
This is also well understood. We wanted to give as much real data as possible in this challenge, but we didn't have a reliable way of generating more than what the LIGO detectors produced during the run, and we did not want to run into issues about having too much correlation among different samples. For this reason, we decided to generate synthetic data using Gaussian noise, for which a PSD (or ASD, or sqrtSX, they are all essentially equivalent in this context) needs to be specified.
\n\nIt would us sidechannel which is not present in real GW search data.
\n
This is not a problem in CW searches. For \"clean\" (relatively quiet) bands we are able to estimate the PSD of the noise with quite good accuracy, and that is sufficient for us to actually perform our analysis. The relevant bit is that, regardless of that PSD, the relevant quantity when it comes to detecting a signal is the SNR (which is roughly a ratio between signal amplitude h0 and noise amplitude ASD). Signals are usually so weak (read \"SNR is low\") that we cannot use PSD as a trigger for signals; in fact, high PSDs are usually related to noise artifacts.
All in all, the test set portraits a data distribution which is a close as possible to a general CW search as we could generate. There is a significant fraction of simulated data, as we state in the data page, and that is due to the impossibility of properly and reliably generating \"real data\" on demand, but that does not drift this competition away from the sort of \"realistic\" setup we want you to focus on.
\nWe are happy to clarify any other concerns, so please keep raising any issues you may be concerned :)
\nCheers,
", "votes": 3, "replies": [ { "id": 2014681, "author_name": "Josef Slavicek", "author_url": "", "post_date": "2022-11-02T18:40:17.017000", "content": "OK, thanks for clarification.
", "votes": 0, "replies": [] } ] }, { "id": 2013238, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-11-01T18:29:38.083000", "content": "\"My guess is, that most of training data was generated with single value of sqrtSX. \"
\nYep. 5e-24 or so.
FYI, I have evaluated using predictions from my NN model and found that most of the samples where the std in complex128 judged outlier are easily true positives.
\n
Getting same std with complex64 is a property of float32
\nhttps://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/361312
Signal contribute to std (total power) and correlated with label, but I don't think it is a strong estimator, and no worry for a leak. Test data also contain real noise, which are far from the train sqrtSX.
", "votes": 1, "replies": [ { "id": 2013747, "author_name": "Josef Slavicek", "author_url": "", "post_date": "2022-11-02T05:38:54.063000", "content": "\n\nSignal contribute to std (total power) and correlated with label
\n
Yes, but if you generate big part of your dataset with just single value of sqrtSX, you get something what IMO you should not have: very good prior knowledge of std(spectrogram) for negative datapoints. You know it up to 3 decimal places. If people around LIGO have similarly good knowledge of noise in their detector, then everything is OK. If they don't, then this competition will be still fair, but not as useful as it could be. It would give us sidechannel which is not present in real GW search data.
Around 80% of test data seems to be affected by this.
", "votes": 1, "replies": [] }, { "id": 2013992, "author_name": "🐢 Jun Koda", "author_url": "", "post_date": "2022-11-02T08:40:47.510000", "content": "I see, I agree with that. We know the statistics of the simulated noise very well, and those 80% are easier than real data.
", "votes": 0, "replies": [] }, { "id": 2014040, "author_name": "DennisSakva", "author_url": "", "post_date": "2022-11-02T09:16:28.113000", "content": "Don't pay too much attention to the train set. Test set is very different.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2013187": "Hello @rodrigotenorio , @michaeljwill , I discovered very surprising property of train data. If I compute std of spectrogram in complex64, I'm getting almost allways the same value of 1.4973568266036865e-22. If I do the same computation in complex128, the values starts do differ - and they are partly useful as indicator of target label:\n\n\nThe image is taken from [notebook](https://www.kaggle.com/code/josefslavicek/g2net-weird-behavior-of-std-spectrogram) which demonstrate the behavior.\n\nMy guess is, that most of training data was generated with single value of sqrtSX. Is this property of data known and considered as OK? For me, it feels almost like target label leak :-)", "2014503": "Hi @josefslavicek , all, thank you very much for publicly starting this thread.\n\nWe (@michaeljwill and myself) had a brief discussion about this, I hope the following clarifies what is going on.\n\ntl;dr This is not a data leak, we understand the reason this is happening and it is expected. Nothing to worry about.\n\n> and they are partly useful as indicator of target label\n\nYes, this is expected: stdev is computed from the data, and very strong signals will deviate the value away from the \"noise-only\" value. This should already tell you how \"strong\" signals in the test set are, by the way :) .\n\n > Around 80% of test data seems to be affected by this\n\nThis is also well understood. We wanted to give as much real data as possible in this challenge, but we didn't have a reliable way of generating more than what the LIGO detectors produced during the run, and we did not want to run into issues about having too much correlation among different samples. For this reason, we decided to generate synthetic data using Gaussian noise, for which a PSD (or ASD, or `sqrtSX`, they are all essentially equivalent in this context) needs to be specified. \n\n> It would us sidechannel which is not present in real GW search data.\n\nThis is not a problem in CW searches. For \"clean\" (relatively quiet) bands we are able to estimate the PSD of the noise with quite good accuracy, and that is sufficient for us to actually perform our analysis. The relevant bit is that, regardless of that PSD, the relevant quantity when it comes to detecting a signal is the SNR (which is roughly a ratio between signal amplitude `h0` and noise amplitude `ASD`). Signals are usually so weak (read \"SNR is low\") that we cannot use PSD as a trigger for signals; in fact, high PSDs are usually related to noise artifacts.\n\nAll in all, the test set portraits a data distribution which is a close as possible to a general CW search as we could generate. There is a significant fraction of simulated data, as we state in the data page, and that is due to the impossibility of properly and reliably generating \"real data\" on demand, but that does not drift this competition away from the sort of \"realistic\" setup we want you to focus on.\n\nWe are happy to clarify any other concerns, so please keep raising any issues you may be concerned :)\n\nCheers,", "2013238": "\"My guess is, that most of training data was generated with single value of sqrtSX. \"\nYep. 5e-24 or so.", "2014400": "FYI, I have evaluated using predictions from my NN model and found that most of the samples where the std in complex128 judged outlier are easily true positives.\n", "2013483": "Getting same std with complex64 is a property of float32\nhttps://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/discussion/361312\n\nSignal contribute to std (total power) and correlated with label, but I don't think it is a strong estimator, and no worry for a leak. Test data also contain real noise, which are far from the train sqrtSX." } }