\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1973217%2Fc82503ad6f32fe8bc11d7981e9f3d350%2FScreen%20Shot%202022-12-12%20at%202.17.10%20PM.png?generation=1672906996452633&alt=media\")
First of all, I would like to express deep gratitude to the competition organisers and the Kaggle team. The competition covers a wide range of knowledge, including data generation, deep learning, and classical methods, making it one of the most challenging and interesting competitions in my own Kaggle history. Without my wonderful teammates this ranking would not have been possible and it was indeed a pleasure for me to work with them.
\nCongratulations to all the winners (and a special big congratulations to the new GM and my amazing teammate Gleb)! It was very interesting to read about various methods used in other teams' solutions.
Our approach is comprised of an ensemble of various deep learning models and \"creative denoising\", which is a denoising technique that exploits the frequency overlap of real-world noise.
\nWe generated 40k images, including stationary noise, non-stationary noise, glitches (horizontal instrument lines) using pyfstat.
\nAs we look into the data, we found that background noise follows a chi square distribution (the real and imaginary parts follow independent gaussian distributions). We therefore considered a fast online generation of noise with almost identical statistics (mean, variance) to each test data (This notebook shows how we generated noise).
\nWe generated 32k to 128k signal templates using pyfstat with sqrtSX=0, and injected signals into the generated noise. Also, since signals has frequency dependency (Figure 1), during training phase we:
In this way, we were able to train our models with almost infinite background noise patterns. This prevented the model from overfitting to the background noise, and improved the model performance significantly.
\nFigure 1.
\n
We generated a validation set using the infinite training approach described above, with a one-to-one correspondence to the test set. Positive ratio was .66, and the injected signals has random parameters with signal depth ranged from 25 to 50. This validation set had good correlation with public LB.
\nWe synchronized the spectrogram from two detectors so that the model can catch the signal much easier, with no need to look around. Temporally averaged bins (128 - 720 bins) were used as the input image.
\n( @chris62 )
\nWe normalized the entire input image by subtracting mean and dividing by the std dev
( @iafoss @analokamus )
\nSince background noise follows chi square distribution, we first calculated sqrt of the input image so that the output should follow normal distribution, and then subtracted mean / divided by std in each column to mitigate the impact of nonstationary noise.
We used CNNs with the following modification:
\nVarious backbones including EfficientNetB7, ResNeXt50, Inception v4, Xception65, Convnext base and ViT were used.
\n| Model name | \nPublic LB | \nPrivate LB | \nAuthor | \n
|---|---|---|---|
| VIT FREQ | \n0.779 | \n0.793 | \n@drhb | \n
- first 2 convolution layers from inception -> image_features\n- [frequency, stat based features ] -> embedding \n- combining [image_features, embedding]\n- applying this to `vit_large_patch16_224_in21k`\n- trained for 30 epoch\n- lr: 1e-4, cosine annealing\n- augmentation: flip, time/freq masking\n| Model name | \nPublic LB | \nPrivate LB | \nAuthor | \n
|---|---|---|---|
| Model03 | \n0.774 | \n0.792 | \n@analokamus | \n
- infinite training with 80k signal templates\n- input image size = 360 x 360\n- 3x31 denoising layer\n- UNet EfficientNet-b7\n- loss = BCE loss x 0.7 + dice loss x 0.3\n- cosine annealing lr for 40 epochs\n- H/V flip, time/freq masking, random amplifier\n( @analokmaus )
\nI randomly sampled 4096 or 8192 signals from the signal bank, and used them as convolution kernels.
\nThe output image with size of 360 x {num filter} is then passed to a simple CNN classification head.
| Model name | \nPublic LB | \nPrivate LB | \nAuthor | \n
|---|---|---|---|
| Model03 | \n0.768 | \n0.782 | \n@analokamus | \n
- infinite training with 80k signal templates\n- input image size = 360 x 360\n- 4096 x 135 x 360 filters\n- EfficientNet-b0 head\n- BCE loss\n- cosine annealing lr for 40 epochs\n- H/V flip, time/freq masking, random amplifier\n( @bakeryproducts ) WIP
\n( @analokmaus @drhb)
\nTo combine tons of predictions, we used the following approach:
The purpose of this two-stage stacking is to deal with multicollinearity and prevent the coefficients from falling to zero. We used two different ensemble profiles, which resulted in public/private score of 0.784/0.780 and 0.783/0.798 respectively.
\n( @iafoss )
\nConsideration of 2048 random 4-fold splits, fitting 8192 linear models with a weak regularization, evaluation of weights of each model as an average over the final weight distribution.
\nThis approch resulted in public/private score of 0.784/0.801.
Our final submission is the rank average of the three ensemble results described above, scoring 0.784 at public LB and 0.801 at private LB. This is equivalent to 5th place without the denoising technique (leak) we will mention below.
\n( @iafoss @analokamus )
\nThis notebook explains our denoising algorithm.
Four days before the end of the competition we realized that the nonstationary noise test data was generated by taking a large single chunk of real detector noise data, adding some white noise, a random selection of a subset of time_ids, sorting them, assigning randomly selected time_stamps, selection of the frequency range, and adding the signal. The frequency also appeared to be shifted by an integer number of Hz.
\nWe went through all nonstationary noise test samples, performed a search between overlapping frequencies as all vs all bin match, found identical bins, and computed abs1 - abs2, which gives a noise-free signal.
\nOne challenge after denoising is that we need to detect the origin of signal from the matched pairs. We summed spectrogram along time axis of the denoised image (abs1 - abs2), then calculated positive area and negative area. Big positive area suggest that the signal is from abs1 and negative area vice versa. Applying an adequate threshold made it possible to detect the origin of signal very accurately (Figure 2).
\nFigure 2. (left: denoised image, right: positive/negative area, top: no signal, middle: one signal, bottom: two signals)
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1973217%2Fe444b75a9363396a58752932747aa88a%2F546913595-b7d0f04b5.png?generation=1672917875266319&alt=media\")
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1973217%2Fb66a526d15fbf0547a1b60b31a211ff9%2F1209123fd-d62fb035d.png?generation=1672917955287888&alt=media\")
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1973217%2Fd209d9808da5996c97f01472895d6128%2F014065fac-73d5e4fc4.png?generation=1672918045232649&alt=media\")
We were able to denoise approximately 1100 samples, though in most cases the frequency coverage is partial. Therefore, the denoised data cannot give 100% confidence regarding the labels. We interpolated the prediction and denoised prediction based on the coverage. Overall this denoising improved our best ensemble from 0.784 to 0.807 at public LB, from 0.801 to 0.826 at private LB.
\n( @bakeryproducts ) WIP
\nFirst of all, I would like to express deep gratitude to the competition organisers and the Kaggle team. The competition covers a wide range of knowledge, including data generation, deep learning, and classical methods, making it one of the most challenging and interesting competitions in my own Kaggle history. Without my wonderful teammates this ranking would not have been possible and it was indeed a pleasure for me to work with them.
\nCongratulations to all the winners (and a special big congratulations to the new GM and my amazing teammate Gleb)! It was very interesting to read about various methods used in other teams' solutions.
Our approach is comprised of an ensemble of various deep learning models and \"creative denoising\", which is a denoising technique that exploits the frequency overlap of real-world noise.
\nWe generated 40k images, including stationary noise, non-stationary noise, glitches (horizontal instrument lines) using pyfstat.
\nAs we look into the data, we found that background noise follows a chi square distribution (the real and imaginary parts follow independent gaussian distributions). We therefore considered a fast online generation of noise with almost identical statistics (mean, variance) to each test data (This notebook shows how we generated noise).
\nWe generated 32k to 128k signal templates using pyfstat with sqrtSX=0, and injected signals into the generated noise. Also, since signals has frequency dependency (Figure 1), during training phase we:
In this way, we were able to train our models with almost infinite background noise patterns. This prevented the model from overfitting to the background noise, and improved the model performance significantly.
\nFigure 1.
\n
We generated a validation set using the infinite training approach described above, with a one-to-one correspondence to the test set. Positive ratio was .66, and the injected signals has random parameters with signal depth ranged from 25 to 50. This validation set had good correlation with public LB.
\nWe synchronized the spectrogram from two detectors so that the model can catch the signal much easier, with no need to look around. Temporally averaged bins (128 - 720 bins) were used as the input image.
\n( @chris62 )
\nWe normalized the entire input image by subtracting mean and dividing by the std dev
( @iafoss @analokamus )
\nSince background noise follows chi square distribution, we first calculated sqrt of the input image so that the output should follow normal distribution, and then subtracted mean / divided by std in each column to mitigate the impact of nonstationary noise.
We used CNNs with the following modification:
\nVarious backbones including EfficientNetB7, ResNeXt50, Inception v4, Xception65, Convnext base and ViT were used.
\n| Model name | \nPublic LB | \nPrivate LB | \nAuthor | \n
|---|---|---|---|
| VIT FREQ | \n0.779 | \n0.793 | \n@drhb | \n
- first 2 convolution layers from inception -> image_features\n- [frequency, stat based features ] -> embedding \n- combining [image_features, embedding]\n- applying this to `vit_large_patch16_224_in21k`\n- trained for 30 epoch\n- lr: 1e-4, cosine annealing\n- augmentation: flip, time/freq masking\n| Model name | \nPublic LB | \nPrivate LB | \nAuthor | \n
|---|---|---|---|
| Model03 | \n0.774 | \n0.792 | \n@analokamus | \n
- infinite training with 80k signal templates\n- input image size = 360 x 360\n- 3x31 denoising layer\n- UNet EfficientNet-b7\n- loss = BCE loss x 0.7 + dice loss x 0.3\n- cosine annealing lr for 40 epochs\n- H/V flip, time/freq masking, random amplifier\n( @analokmaus )
\nI randomly sampled 4096 or 8192 signals from the signal bank, and used them as convolution kernels.
\nThe output image with size of 360 x {num filter} is then passed to a simple CNN classification head.
| Model name | \nPublic LB | \nPrivate LB | \nAuthor | \n
|---|---|---|---|
| Model03 | \n0.768 | \n0.782 | \n@analokamus | \n
- infinite training with 80k signal templates\n- input image size = 360 x 360\n- 4096 x 135 x 360 filters\n- EfficientNet-b0 head\n- BCE loss\n- cosine annealing lr for 40 epochs\n- H/V flip, time/freq masking, random amplifier\n( @bakeryproducts ) WIP
\n( @analokmaus @drhb)
\nTo combine tons of predictions, we used the following approach:
The purpose of this two-stage stacking is to deal with multicollinearity and prevent the coefficients from falling to zero. We used two different ensemble profiles, which resulted in public/private score of 0.784/0.780 and 0.783/0.798 respectively.
\n( @iafoss )
\nConsideration of 2048 random 4-fold splits, fitting 8192 linear models with a weak regularization, evaluation of weights of each model as an average over the final weight distribution.
\nThis approch resulted in public/private score of 0.784/0.801.
Our final submission is the rank average of the three ensemble results described above, scoring 0.784 at public LB and 0.801 at private LB. This is equivalent to 5th place without the denoising technique (leak) we will mention below.
\n( @iafoss @analokamus )
\nThis notebook explains our denoising algorithm.
Four days before the end of the competition we realized that the nonstationary noise test data was generated by taking a large single chunk of real detector noise data, adding some white noise, a random selection of a subset of time_ids, sorting them, assigning randomly selected time_stamps, selection of the frequency range, and adding the signal. The frequency also appeared to be shifted by an integer number of Hz.
\nWe went through all nonstationary noise test samples, performed a search between overlapping frequencies as all vs all bin match, found identical bins, and computed abs1 - abs2, which gives a noise-free signal.
\nOne challenge after denoising is that we need to detect the origin of signal from the matched pairs. We summed spectrogram along time axis of the denoised image (abs1 - abs2), then calculated positive area and negative area. Big positive area suggest that the signal is from abs1 and negative area vice versa. Applying an adequate threshold made it possible to detect the origin of signal very accurately (Figure 2).
\nFigure 2. (left: denoised image, right: positive/negative area, top: no signal, middle: one signal, bottom: two signals)
\n
\n
\n
We were able to denoise approximately 1100 samples, though in most cases the frequency coverage is partial. Therefore, the denoised data cannot give 100% confidence regarding the labels. We interpolated the prediction and denoised prediction based on the coverage. Overall this denoising improved our best ensemble from 0.784 to 0.807 at public LB, from 0.801 to 0.826 at private LB.
\n( @bakeryproducts ) WIP
\n\n\n## Validation stratedy\nWe generated a validation set using the infinite training approach described above, with a one-to-one correspondence to the test set. Positive ratio was .66, and the injected signals has random parameters with signal depth ranged from 25 to 50. This validation set had good correlation with public LB. \n\n## Preprocessing \n### Temporal synchronization and resize\nWe synchronized the spectrogram from two detectors so that the model can catch the signal much easier, with no need to look around. Temporally averaged bins (128 - 720 bins) were used as the input image.\n\n### Spectrogram normalization\n#### Global normalization\n( @chris62 )\nWe normalized the entire input image by subtracting mean and dividing by the std dev\n\n#### Column-wise sqrt normalization\n( @iafoss @analokamus )\nSince background noise follows chi square distribution, we first calculated sqrt of the input image so that the output should follow normal distribution, and then subtracted mean / divided by std in each column to mitigate the impact of nonstationary noise.\n\n## Architectures and training\n### Spectrogram image classifier\nWe used CNNs with the following modification:\n- Denoising layer(s) in the beginning with large kernels (3x31, 5x15, 5x31, 7x65)\n- AE denoising based pretraining for large kernel layers ( @Iafoss )\n- Denoising AUX with a light Segformer style decoder and MSE loss for noise-free signal ( @Iafoss )\n- Add a UNet segmentation decoder to extract signal and use dice loss as aux loss ( @analokamus )\n- Signal depth scheduling: start with 10-20 depth and finish with 25-50 depth ( @Iafoss @analokamus )\n- Add frequency to the output head ( @chris62 )\n- Incorporate frequency information and statistics based features ( @drhb )\n\nVarious backbones including EfficientNetB7, ResNeXt50, Inception v4, Xception65, Convnext base and ViT were used. \n\n| Model name | Public LB | Private LB | Author |\n| --- | --- | --- | --- |\n| VIT FREQ | **0.779** | **0.793** | @drhb |\n```\n- first 2 convolution layers from inception -> image_features\n- [frequency, stat based features ] -> embedding \n- combining [image_features, embedding]\n- applying this to `vit_large_patch16_224_in21k`\n- trained for 30 epoch\n- lr: 1e-4, cosine annealing\n- augmentation: flip, time/freq masking\n``` \n\n| Model name | Public LB | Private LB | Author |\n| --- | --- | --- | --- |\n| Model03 | 0.774 | 0.792 | @analokamus |\n```\n- infinite training with 80k signal templates\n- input image size = 360 x 360\n- 3x31 denoising layer\n- UNet EfficientNet-b7\n- loss = BCE loss x 0.7 + dice loss x 0.3\n- cosine annealing lr for 40 epochs\n- H/V flip, time/freq masking, random amplifier\n``` \n\n### Matched filter-like classifier\n( @analokmaus )\nI randomly sampled 4096 or 8192 signals from the signal bank, and used them as convolution kernels. \nThe output image with size of 360 x {num filter} is then passed to a simple CNN classification head.\n\n| Model name | Public LB | Private LB | Author |\n| --- | --- | --- | --- |\n| Model03 | 0.768 | 0.782 | @analokamus |\n```\n- infinite training with 80k signal templates\n- input image size = 360 x 360\n- 4096 x 135 x 360 filters\n- EfficientNet-b0 head\n- BCE loss\n- cosine annealing lr for 40 epochs\n- H/V flip, time/freq masking, random amplifier\n``` \n\n### Megaconv\n( @bakeryproducts ) WIP\n\n## Ensemble\n### Hierarchical stacking\n( @analokmaus @drhb)\nTo combine tons of predictions, we used the following approach: \n1. Separate predictions into stationary and nonstationary, this is because we observed different correlation patterns between predictions in noise types.\n2. Categorize predictions into some groups based on the similarity of their training settings and fit a ridge regression model (large alpha regularization, positive coefficients only) to maximize AUC of the validation set inside each group (first stage stacking).\n3. Collect results from first stage stacking (shaped {num sample} x {num group}), and fit another ridge regression (large alpha regularization, positive coefficients only) to maximize AUC of the validation set. \n4. Do 2-3 for both noise types defined in 1\n\nThe purpose of this two-stage stacking is to deal with multicollinearity and prevent the coefficients from falling to zero. We used two different ensemble profiles, which resulted in public/private score of 0.784/0.780 and 0.783/0.798 respectively.\n\n### Bayesian-like approach\n( @iafoss )\nConsideration of 2048 random 4-fold splits, fitting 8192 linear models with a weak regularization, evaluation of weights of each model as an average over the final weight distribution.\nThis approch resulted in public/private score of 0.784/0.801.\n\nOur final submission is the rank average of the three ensemble results described above, scoring **0.784 at public LB and 0.801 at private LB**. This is equivalent to **5th place** without the denoising technique (leak) we will mention below.\n\n## Denoising (or leak 😉?)\n### Internal denoising\n( @iafoss @analokamus )\n[This notebook](https://www.kaggle.com/code/iafoss/creative-denoising-part-of-top-3-solution/notebook) explains our denoising algorithm.\n\nFour days before the end of the competition we realized that the nonstationary noise test data was generated by taking a large single chunk of real detector noise data, adding some white noise, a random selection of a subset of time_ids, sorting them, assigning randomly selected time_stamps, selection of the frequency range, and adding the signal. The frequency also appeared to be shifted by an integer number of Hz.\n\nWe went through all nonstationary noise test samples, performed a search between overlapping frequencies as all vs all bin match, found identical bins, and computed abs1 - abs2, which gives a noise-free signal. \n\nOne challenge after denoising is that we need to detect the origin of signal from the matched pairs. We summed spectrogram along time axis of the denoised image (abs1 - abs2), then calculated positive area and negative area. Big positive area suggest that the signal is from abs1 and negative area vice versa. Applying an adequate threshold made it possible to detect the origin of signal very accurately (**Figure 2**).\n\n**Figure 2**. (left: denoised image, right: positive/negative area, top: no signal, middle: one signal, bottom: two signals)\n\n\n\n\nWe were able to denoise approximately 1100 samples, though in most cases the frequency coverage is partial. Therefore, the denoised data cannot give 100% confidence regarding the labels. We interpolated the prediction and denoised prediction based on the coverage. Overall this denoising improved our best ensemble from 0.784 to 0.807 at public LB, from 0.801 to **0.826** at private LB. \n\n### External denoising\n( @bakeryproducts ) WIP\n\n## Things did not work\n- Noise to Noise denoising\n- Noise to Void denoising\n- Diffusion models\n- Advanced stacking of the models with adding extra features",
"votes": 40
},
{
"id": 2087206,
"postDate": "2023-01-05T12:22:06.647Z",
"content": "Congrats with the third place! This is an astonishing amount of work!
", "rawMarkdown": "Congrats with the third place! This is an astonishing amount of work!", "votes": 3, "replies": [ { "id": 2087229, "postDate": "2023-01-05T12:56:17.440Z", "content": "Thank you! Your team's solution, backed by solid domain knowledge, was very elegant and informative. It was an honour to compete with you.
", "rawMarkdown": "Thank you! Your team's solution, backed by solid domain knowledge, was very elegant and informative. It was an honour to compete with you.", "votes": 2, "replies": [ { "id": 2087342, "postDate": "2023-01-05T14:37:27.363Z", "content": "Thank you for your kind words, we feel the same way.
", "rawMarkdown": "Thank you for your kind words, we feel the same way.", "votes": 2 } ] } ] }, { "id": 2087351, "postDate": "2023-01-05T14:43:55.493Z", "content": "Congratulations on your gold medal!! Awesome Laura Romar! Thank you for sharing. It's very helpful for us to learn.
", "rawMarkdown": "Congratulations on your gold medal!! Awesome Laura Romar! Thank you for sharing. It's very helpful for us to learn.", "votes": 1 }, { "id": 2087611, "postDate": "2023-01-05T18:02:47.057Z", "content": "Cool, I also subtracted overlapping test images after finding bounding boxes. I had to deal with the boundary between that was now much higher by renormalizing that. But somehow this only made my AUC worse. From what I saw, the match still wasn't perfect, because (like you said) there is added noise. But the non-stationary part and any glitches seems to match. It does drop the noise alot in most cases where alot of overlap occurs. Unsure why it didn't help me.
\nI only did this 2 days before competition ended, so possibly I could have gone further like you guys. Great job.
", "rawMarkdown": "Cool, I also subtracted overlapping test images after finding bounding boxes. I had to deal with the boundary between that was now much higher by renormalizing that. But somehow this only made my AUC worse. From what I saw, the match still wasn't perfect, because (like you said) there is added noise. But the non-stationary part and any glitches seems to match. It does drop the noise alot in most cases where alot of overlap occurs. Unsure why it didn't help me.\n\nI only did this 2 days before competition ended, so possibly I could have gone further like you guys. Great job.", "votes": 2, "replies": [ { "id": 2088139, "postDate": "2023-01-06T05:22:54.103Z", "content": "Thank you for the comment! This notebook shows how we denoised the spectrogram.
\nhttps://www.kaggle.com/code/iafoss/creative-denoising-part-of-top-3-solution/notebook
Congratulations! You have pushed deep learning solution to a new level
", "rawMarkdown": "Congratulations! You have pushed deep learning solution to a new level", "votes": 2 }, { "id": 2087339, "postDate": "2023-01-05T14:36:04.223Z", "content": "Amazing solution! It is very interesting to see how different your deep learning solution is from 1st place's no machine learning approach. Great job and congrats to the team!
", "rawMarkdown": "Amazing solution! It is very interesting to see how different your deep learning solution is from 1st place's no machine learning approach. Great job and congrats to the team!" } ], "comments": [ { "id": 2087206, "author_name": "Inar Timiryasov", "author_url": "", "post_date": "2023-01-05T12:22:06.647000", "content": "Congrats with the third place! This is an astonishing amount of work!
", "votes": 3, "replies": [ { "id": 2087229, "author_name": "RabotniKuma", "author_url": "", "post_date": "2023-01-05T12:56:17.440000", "content": "Thank you! Your team's solution, backed by solid domain knowledge, was very elegant and informative. It was an honour to compete with you.
", "votes": 2, "replies": [ { "id": 2087342, "author_name": "Inar Timiryasov", "author_url": "", "post_date": "2023-01-05T14:37:27.363000", "content": "Thank you for your kind words, we feel the same way.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2087351, "author_name": "BarryZhou", "author_url": "", "post_date": "2023-01-05T14:43:55.493000", "content": "Congratulations on your gold medal!! Awesome Laura Romar! Thank you for sharing. It's very helpful for us to learn.
", "votes": 1, "replies": [] }, { "id": 2087611, "author_name": "Jonathan McKinney", "author_url": "", "post_date": "2023-01-05T18:02:47.057000", "content": "Cool, I also subtracted overlapping test images after finding bounding boxes. I had to deal with the boundary between that was now much higher by renormalizing that. But somehow this only made my AUC worse. From what I saw, the match still wasn't perfect, because (like you said) there is added noise. But the non-stationary part and any glitches seems to match. It does drop the noise alot in most cases where alot of overlap occurs. Unsure why it didn't help me.
\nI only did this 2 days before competition ended, so possibly I could have gone further like you guys. Great job.
", "votes": 2, "replies": [ { "id": 2088139, "author_name": "RabotniKuma", "author_url": "", "post_date": "2023-01-06T05:22:54.103000", "content": "Thank you for the comment! This notebook shows how we denoised the spectrogram.
\nhttps://www.kaggle.com/code/iafoss/creative-denoising-part-of-top-3-solution/notebook
Congratulations! You have pushed deep learning solution to a new level
", "votes": 2, "replies": [] }, { "id": 2087339, "author_name": "Ravi Shah", "author_url": "", "post_date": "2023-01-05T14:36:04.223000", "content": "Amazing solution! It is very interesting to see how different your deep learning solution is from 1st place's no machine learning approach. Great job and congrats to the team!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2087168": "First of all, I would like to express deep gratitude to the competition organisers and the Kaggle team. The competition covers a wide range of knowledge, including data generation, deep learning, and classical methods, making it one of the most challenging and interesting competitions in my own Kaggle history. Without my wonderful teammates this ranking would not have been possible and it was indeed a pleasure for me to work with them.\nCongratulations to all the winners (and a special big congratulations to the new GM and my amazing teammate Gleb)! It was very interesting to read about various methods used in other teams' solutions.\n\nOur approach is comprised of an ensemble of various **deep learning** models and **\"creative denoising\"**, which is a denoising technique that exploits the frequency overlap of real-world noise.\n\n## Data generation\n### Fixed dataset approach\nWe generated 40k images, including stationary noise, non-stationary noise, glitches (horizontal instrument lines) using pyfstat.\n\n### Infinite training approach\nAs we look into the data, we found that background noise follows a chi square distribution (the real and imaginary parts follow independent gaussian distributions). We therefore considered a fast online generation of noise with almost identical statistics (mean, variance) to each test data ([This notebook](https://www.kaggle.com/code/analokamus/fast-online-noise-generation/notebook?scriptVersionId=115544433) shows how we generated noise). \nWe generated 32k to 128k signal templates using pyfstat with sqrtSX=0, and injected signals into the generated noise. Also, since signals has frequency dependency (**Figure 1**), during training phase we: \n1. Sample a test image\n2. Generate noise based on its statistics\n3. Sample a signal generated at the same frequency from signal banks and inject\n\nIn this way, we were able to train our models with almost infinite background noise patterns. This prevented the model from overfitting to the background noise, and improved the model performance significantly.\n\n**Figure 1.**\n\n\n## Validation stratedy\nWe generated a validation set using the infinite training approach described above, with a one-to-one correspondence to the test set. Positive ratio was .66, and the injected signals has random parameters with signal depth ranged from 25 to 50. This validation set had good correlation with public LB. \n\n## Preprocessing \n### Temporal synchronization and resize\nWe synchronized the spectrogram from two detectors so that the model can catch the signal much easier, with no need to look around. Temporally averaged bins (128 - 720 bins) were used as the input image.\n\n### Spectrogram normalization\n#### Global normalization\n( @chris62 )\nWe normalized the entire input image by subtracting mean and dividing by the std dev\n\n#### Column-wise sqrt normalization\n( @iafoss @analokamus )\nSince background noise follows chi square distribution, we first calculated sqrt of the input image so that the output should follow normal distribution, and then subtracted mean / divided by std in each column to mitigate the impact of nonstationary noise.\n\n## Architectures and training\n### Spectrogram image classifier\nWe used CNNs with the following modification:\n- Denoising layer(s) in the beginning with large kernels (3x31, 5x15, 5x31, 7x65)\n- AE denoising based pretraining for large kernel layers ( @Iafoss )\n- Denoising AUX with a light Segformer style decoder and MSE loss for noise-free signal ( @Iafoss )\n- Add a UNet segmentation decoder to extract signal and use dice loss as aux loss ( @analokamus )\n- Signal depth scheduling: start with 10-20 depth and finish with 25-50 depth ( @Iafoss @analokamus )\n- Add frequency to the output head ( @chris62 )\n- Incorporate frequency information and statistics based features ( @drhb )\n\nVarious backbones including EfficientNetB7, ResNeXt50, Inception v4, Xception65, Convnext base and ViT were used. \n\n| Model name | Public LB | Private LB | Author |\n| --- | --- | --- | --- |\n| VIT FREQ | **0.779** | **0.793** | @drhb |\n```\n- first 2 convolution layers from inception -> image_features\n- [frequency, stat based features ] -> embedding \n- combining [image_features, embedding]\n- applying this to `vit_large_patch16_224_in21k`\n- trained for 30 epoch\n- lr: 1e-4, cosine annealing\n- augmentation: flip, time/freq masking\n``` \n\n| Model name | Public LB | Private LB | Author |\n| --- | --- | --- | --- |\n| Model03 | 0.774 | 0.792 | @analokamus |\n```\n- infinite training with 80k signal templates\n- input image size = 360 x 360\n- 3x31 denoising layer\n- UNet EfficientNet-b7\n- loss = BCE loss x 0.7 + dice loss x 0.3\n- cosine annealing lr for 40 epochs\n- H/V flip, time/freq masking, random amplifier\n``` \n\n### Matched filter-like classifier\n( @analokmaus )\nI randomly sampled 4096 or 8192 signals from the signal bank, and used them as convolution kernels. \nThe output image with size of 360 x {num filter} is then passed to a simple CNN classification head.\n\n| Model name | Public LB | Private LB | Author |\n| --- | --- | --- | --- |\n| Model03 | 0.768 | 0.782 | @analokamus |\n```\n- infinite training with 80k signal templates\n- input image size = 360 x 360\n- 4096 x 135 x 360 filters\n- EfficientNet-b0 head\n- BCE loss\n- cosine annealing lr for 40 epochs\n- H/V flip, time/freq masking, random amplifier\n``` \n\n### Megaconv\n( @bakeryproducts ) WIP\n\n## Ensemble\n### Hierarchical stacking\n( @analokmaus @drhb)\nTo combine tons of predictions, we used the following approach: \n1. Separate predictions into stationary and nonstationary, this is because we observed different correlation patterns between predictions in noise types.\n2. Categorize predictions into some groups based on the similarity of their training settings and fit a ridge regression model (large alpha regularization, positive coefficients only) to maximize AUC of the validation set inside each group (first stage stacking).\n3. Collect results from first stage stacking (shaped {num sample} x {num group}), and fit another ridge regression (large alpha regularization, positive coefficients only) to maximize AUC of the validation set. \n4. Do 2-3 for both noise types defined in 1\n\nThe purpose of this two-stage stacking is to deal with multicollinearity and prevent the coefficients from falling to zero. We used two different ensemble profiles, which resulted in public/private score of 0.784/0.780 and 0.783/0.798 respectively.\n\n### Bayesian-like approach\n( @iafoss )\nConsideration of 2048 random 4-fold splits, fitting 8192 linear models with a weak regularization, evaluation of weights of each model as an average over the final weight distribution.\nThis approch resulted in public/private score of 0.784/0.801.\n\nOur final submission is the rank average of the three ensemble results described above, scoring **0.784 at public LB and 0.801 at private LB**. This is equivalent to **5th place** without the denoising technique (leak) we will mention below.\n\n## Denoising (or leak 😉?)\n### Internal denoising\n( @iafoss @analokamus )\n[This notebook](https://www.kaggle.com/code/iafoss/creative-denoising-part-of-top-3-solution/notebook) explains our denoising algorithm.\n\nFour days before the end of the competition we realized that the nonstationary noise test data was generated by taking a large single chunk of real detector noise data, adding some white noise, a random selection of a subset of time_ids, sorting them, assigning randomly selected time_stamps, selection of the frequency range, and adding the signal. The frequency also appeared to be shifted by an integer number of Hz.\n\nWe went through all nonstationary noise test samples, performed a search between overlapping frequencies as all vs all bin match, found identical bins, and computed abs1 - abs2, which gives a noise-free signal. \n\nOne challenge after denoising is that we need to detect the origin of signal from the matched pairs. We summed spectrogram along time axis of the denoised image (abs1 - abs2), then calculated positive area and negative area. Big positive area suggest that the signal is from abs1 and negative area vice versa. Applying an adequate threshold made it possible to detect the origin of signal very accurately (**Figure 2**).\n\n**Figure 2**. (left: denoised image, right: positive/negative area, top: no signal, middle: one signal, bottom: two signals)\n\n\n\n\nWe were able to denoise approximately 1100 samples, though in most cases the frequency coverage is partial. Therefore, the denoised data cannot give 100% confidence regarding the labels. We interpolated the prediction and denoised prediction based on the coverage. Overall this denoising improved our best ensemble from 0.784 to 0.807 at public LB, from 0.801 to **0.826** at private LB. \n\n### External denoising\n( @bakeryproducts ) WIP\n\n## Things did not work\n- Noise to Noise denoising\n- Noise to Void denoising\n- Diffusion models\n- Advanced stacking of the models with adding extra features",
"2087206": "Congrats with the third place! This is an astonishing amount of work!",
"2087351": "Congratulations on your gold medal!! Awesome Laura Romar! Thank you for sharing. It's very helpful for us to learn.",
"2087611": "Cool, I also subtracted overlapping test images after finding bounding boxes. I had to deal with the boundary between that was now much higher by renormalizing that. But somehow this only made my AUC worse. From what I saw, the match still wasn't perfect, because (like you said) there is added noise. But the non-stationary part and any glitches seems to match. It does drop the noise alot in most cases where alot of overlap occurs. Unsure why it didn't help me.\n\nI only did this 2 days before competition ended, so possibly I could have gone further like you guys. Great job.",
"2087269": "Congratulations! You have pushed deep learning solution to a new level",
"2087339": "Amazing solution! It is very interesting to see how different your deep learning solution is from 1st place's no machine learning approach. Great job and congrats to the team!"
}
}