{
  "id": 363280,
  "title": " Recap of the Top Solutions from the Previous G2Net Competition",
  "url": "/competitions/g2net-detecting-continuous-gravitational-waves/discussion/363280",
  "author_name": "Sinan Calisir",
  "post_date": "2022-10-31T23:20:12.091000",
  "votes": 35,
  "comment_count": 5,
  "views": 0,
  "content": "<ul>\n<li>[1st Place Solution] <a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275476\" target=\"_blank\">Part 1</a> <a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275507\" target=\"_blank\">Part 2</a><ul>\n<li>Synthetic data generation!!!</li>\n<li>Conv1D</li>\n<li>optimizer: SGD, wd=1e-4, nesterov momentum, learning rate: 0.1 with cosine annealing</li>\n<li>input: 3 channels of raw signal filtered with butterworth filter at 20hz</li>\n<li>augmentations: freq masking, time masking, channel shuffle (only Hanford, Livingston), minor time shifts between channels, 5ms</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275341\" target=\"_blank\">2nd Place Solution</a><ul>\n<li>Trainable frontend in general outperformed fixed frontend.</li>\n<li>1D CNN, WaveNet wavegram, CWT.</li>\n<li>Applied bandpass filter to all networks. [16, 512] for CWT-CNN and Trainable frontend CNN, [30, 300] for 1d-CNN. Whitening did not work.</li>\n<li>Augmentations: gaussian noise for 2d-CNN networks, and flipped wave amplitude for 1d-CNN.</li>\n<li>Re-training on soft(continuous) pseudo-labelled test dataset improved AUC by ~0.001. Label smoothing during pseudo-label also helped a bit.</li>\n<li>Cross validated Ridge regression model was used to combine the outputs from neural network models.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275617\" target=\"_blank\">3rd Place Solution</a><ul>\n<li>Pretraining with GW, Training with Pseudo Label or Soft Label, BCE, Rank Loss, AdamW, RangerLars+Lookahead optimizer</li>\n<li>Preprocessing:<ul>\n<li>Avg PSD of target 0 (Design Curves)</li>\n<li>Extending waves</li>\n<li>Whitening with Tukey window</li></ul></li>\n<li>2D models:<ul>\n<li>EfficientNet(B3, B4, B5, B7), EfficientNetV2(M), ResNet200D, Inception-V3 (also we performed a number of initial experiments with ResNeXt models)</li>\n<li>CQT and CWT images generated based on the whitened signal</li>\n<li>Image size 128 x 128 〜 512 x 512</li>\n<li>Soft leak-free pseudo labeling from ensemble results</li>\n<li>LIGO channel swap argumentation (randomly swapping LIGO channels) for both training and TTA</li>\n<li>1D mixup prior to CQT/CWT</li>\n<li>Adding a 4th channel to the spectrogram/scalogram input which is just a linear gradient (-1, 1) along the frequency dimension used for frequency encoding (similar to positional encoding in transformers)</li></ul></li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275331\" target=\"_blank\">4th Place Solution</a><ul>\n<li>GW waves numpy array --&gt; horizontal stacking all three to get (1,4096*3) array --&gt; band pass filtering ---&gt; Deep conv1d backbone with residuals ---&gt;LSTM head ---&gt; Prediction</li>\n<li>Numpy ---&gt; signal tukey ---&gt; band pass filter ---&gt; normalized by norm_by =[7.729773e-21,8.228142e-21, 8.750003e-21] ---&gt;CQT--&gt; Augmentations[coloredNoise and shift]</li>\n<li>Colored noise augmentation was done channel wise while shift was applied sample wise.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275334\" target=\"_blank\">5th Place Solution</a><ul>\n<li>PSD whitening with precomputed mean PSD for each channel, custom window function : mirrored sigmoid</li>\n<li>1d augs: shift, noise, mixup, cutmix, shuffle</li>\n<li>2d CQT default settings fmin=16, fmax=1024, hop=12</li>\n<li>5xCNN 2d ensemble with TTA, models and EMA's</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275343\" target=\"_blank\">6th Place Solution</a><ul>\n<li>wave -&gt; dct -&gt; trainable bp filter -&gt; idct -&gt; 1dcnn/cwt -&gt; 1dcnn/2dcnn/resnet/effnetv2/lstm</li>\n<li>best single model：4096x3 -&gt; 1dcnn -&gt; 512x256x3 -&gt; resnet34</li>\n<li>augmentation: random shift wave separately, up to 1/32 second, random change phase</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275349\" target=\"_blank\">7th Place Solution</a><ul>\n<li>Efficient B3 (size=128*128), Wavenet + GRU, Wavenet</li>\n<li>Preprocess: 30Hz highpass filtering, normalized by the absolute means of individual observations</li>\n<li>Augmentation: Roll +/-500 points (p=0.5), Scale 0.85-1.15 (p=0.5), Multiply -1 (p=0.5)</li></ul></li>\n</ul>\n<pre><code>    \n     = L.Input(shape=(, )) \n    x = tf.reshape(,(-,,)) \n    x = wavenet(x, dim, dilations=, kernel_size=)\n    x = L.Dense(dim//)(x)\n    x = wavenet(x, dim, dilations=, kernel_size=)\n    x = L.Dense(dim//)(x)\n    x = L.Dense(dim)(x)\n    x = tf.reshape(x, (-,,,dim))\n    x = tf.transpose(x,(,,,))\n    x = L.BatchNormalization()(x)\n    x = L.Activation()(x)\n</code></pre>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275433\" target=\"_blank\">8th Place Solution</a><ul>\n<li>Data was multiplied by 1e19 (1e21 later) to ensure that CQT and CWT could be performed correctly using FP16.</li>\n<li>Input images: used nnAudio CQT (or CWT) as the first model layer. Then 3 outputs are stacked on frequency axis. Then divided the image by NEG (average of negative sample images) to get rid of noise on average. Finally applied a log scaling. Depending on the model added the mask as an additional channel. Then scaled data to have multiple of 16 dimensions.</li>\n<li>Mostly used efficientnet_v2s_s and efficientnet_v2_m from timm package.</li>\n<li>Used BCEWithLogitsLoss, Adam and OneCycleLR.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275353\" target=\"_blank\">10th Place Solution</a><ul>\n<li>CWT + Resnet34/EfNet, 488x512, 1/5 fold, train 5 epochs</li>\n<li>Replace CWT with a multiple layers 1D Conv, the same size as CWT, train 20 epochs</li>\n<li>Using different parameter sets(bandpass, time shift, random channel off etc.) to train 10 models above. Ensemble them all.</li>\n<li>Augmentation<ul>\n<li>Trim the original wave to (3, 3904), Random shift +- 65</li>\n<li>Random shift each channel +- 5</li>\n<li>Random turn off one channel</li>\n<li>Random switch channel of two waves (target=0 only)</li></ul></li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275405\" target=\"_blank\">11th Place Solution</a><ul>\n<li>preprocessing: bpf 30-500, bpf 25-1020</li>\n<li>augmentation: mixup</li>\n<li>1DCNN models: used residual connection and dilated convolution/standard convolution with ~24 layers, ~2M parameters.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275356\" target=\"_blank\">12nd Place Solution</a><ul>\n<li>Divide the all wave form by 4.6152116213830774e-20 (max(abs) of the entire train and test data).</li>\n<li>Use nnAudio to run CQT. Randomly select one of flattop, blackmanharris, or nuttall to the window.</li>\n<li>The spectrograms are combined in the frequency direction and input to the model as a single-channel image</li>\n<li>Resize to 384x512 before inputting into the model, and normalize the spectrogram of the entire data set in mean, std</li>\n<li>Augmentation: Before converting to a spectrogram, mixup. Randomly roll shift in time direction</li>\n<li>model: timm tf_efficientnet_b4_ap, Adam, CosineAnnealingLR</li>\n<li>Just one part of the final solution. Lots of models in the final stacking!</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275841\" target=\"_blank\">13th Place Solution</a><ul>\n<li>3 Channel (native) - stacked the detectors channel wise.</li>\n<li>6 Permutations - Horizontally stacked the 3 detectors with different permutations. Then stacked these 6 permutations channel wise.</li>\n<li>CQT+CWT - Took 3 channels(for 3 detectors) from CQT transformation and 3 from CWT transformation and stacked them.</li>\n<li>Double CWT - created two parallel CWT transformations with different parameters</li>\n<li>Stacking</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275431\" target=\"_blank\">15th Place Solution</a><ul>\n<li>Conv2D</li>\n<li>Whitening: Using average PSD. Averaging over all noise samples for each site.</li>\n<li>CQT Scaling with filter_scale = 8/bins_per_octave and (fmin, fmax)=(20, 1024). Both abs and angle part were used.</li>\n<li>Augmentation<ul>\n<li>Horizontal/time shift</li>\n<li>Pad both side and then horizontal random crop to get time shift image.</li>\n<li>Mixup, prevent from overfitting</li>\n<li>GeM Fixed power 3 was better than the trainable case.</li></ul></li></ul></li>\n</ul>",
  "messages": [
    {
      "id": 2011892,
      "postDate": "2022-10-31T23:20:12.093Z",
      "content": "<ul>\n<li>[1st Place Solution] <a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275476\" target=\"_blank\">Part 1</a> <a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275507\" target=\"_blank\">Part 2</a><ul>\n<li>Synthetic data generation!!!</li>\n<li>Conv1D</li>\n<li>optimizer: SGD, wd=1e-4, nesterov momentum, learning rate: 0.1 with cosine annealing</li>\n<li>input: 3 channels of raw signal filtered with butterworth filter at 20hz</li>\n<li>augmentations: freq masking, time masking, channel shuffle (only Hanford, Livingston), minor time shifts between channels, 5ms</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275341\" target=\"_blank\">2nd Place Solution</a><ul>\n<li>Trainable frontend in general outperformed fixed frontend.</li>\n<li>1D CNN, WaveNet wavegram, CWT.</li>\n<li>Applied bandpass filter to all networks. [16, 512] for CWT-CNN and Trainable frontend CNN, [30, 300] for 1d-CNN. Whitening did not work.</li>\n<li>Augmentations: gaussian noise for 2d-CNN networks, and flipped wave amplitude for 1d-CNN.</li>\n<li>Re-training on soft(continuous) pseudo-labelled test dataset improved AUC by ~0.001. Label smoothing during pseudo-label also helped a bit.</li>\n<li>Cross validated Ridge regression model was used to combine the outputs from neural network models.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275617\" target=\"_blank\">3rd Place Solution</a><ul>\n<li>Pretraining with GW, Training with Pseudo Label or Soft Label, BCE, Rank Loss, AdamW, RangerLars+Lookahead optimizer</li>\n<li>Preprocessing:<ul>\n<li>Avg PSD of target 0 (Design Curves)</li>\n<li>Extending waves</li>\n<li>Whitening with Tukey window</li></ul></li>\n<li>2D models:<ul>\n<li>EfficientNet(B3, B4, B5, B7), EfficientNetV2(M), ResNet200D, Inception-V3 (also we performed a number of initial experiments with ResNeXt models)</li>\n<li>CQT and CWT images generated based on the whitened signal</li>\n<li>Image size 128 x 128 〜 512 x 512</li>\n<li>Soft leak-free pseudo labeling from ensemble results</li>\n<li>LIGO channel swap argumentation (randomly swapping LIGO channels) for both training and TTA</li>\n<li>1D mixup prior to CQT/CWT</li>\n<li>Adding a 4th channel to the spectrogram/scalogram input which is just a linear gradient (-1, 1) along the frequency dimension used for frequency encoding (similar to positional encoding in transformers)</li></ul></li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275331\" target=\"_blank\">4th Place Solution</a><ul>\n<li>GW waves numpy array --&gt; horizontal stacking all three to get (1,4096*3) array --&gt; band pass filtering ---&gt; Deep conv1d backbone with residuals ---&gt;LSTM head ---&gt; Prediction</li>\n<li>Numpy ---&gt; signal tukey ---&gt; band pass filter ---&gt; normalized by norm_by =[7.729773e-21,8.228142e-21, 8.750003e-21] ---&gt;CQT--&gt; Augmentations[coloredNoise and shift]</li>\n<li>Colored noise augmentation was done channel wise while shift was applied sample wise.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275334\" target=\"_blank\">5th Place Solution</a><ul>\n<li>PSD whitening with precomputed mean PSD for each channel, custom window function : mirrored sigmoid</li>\n<li>1d augs: shift, noise, mixup, cutmix, shuffle</li>\n<li>2d CQT default settings fmin=16, fmax=1024, hop=12</li>\n<li>5xCNN 2d ensemble with TTA, models and EMA's</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275343\" target=\"_blank\">6th Place Solution</a><ul>\n<li>wave -&gt; dct -&gt; trainable bp filter -&gt; idct -&gt; 1dcnn/cwt -&gt; 1dcnn/2dcnn/resnet/effnetv2/lstm</li>\n<li>best single model：4096x3 -&gt; 1dcnn -&gt; 512x256x3 -&gt; resnet34</li>\n<li>augmentation: random shift wave separately, up to 1/32 second, random change phase</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275349\" target=\"_blank\">7th Place Solution</a><ul>\n<li>Efficient B3 (size=128*128), Wavenet + GRU, Wavenet</li>\n<li>Preprocess: 30Hz highpass filtering, normalized by the absolute means of individual observations</li>\n<li>Augmentation: Roll +/-500 points (p=0.5), Scale 0.85-1.15 (p=0.5), Multiply -1 (p=0.5)</li></ul></li>\n</ul>\n<pre><code>    \n     = L.Input(shape=(, )) \n    x = tf.reshape(,(-,,)) \n    x = wavenet(x, dim, dilations=, kernel_size=)\n    x = L.Dense(dim//)(x)\n    x = wavenet(x, dim, dilations=, kernel_size=)\n    x = L.Dense(dim//)(x)\n    x = L.Dense(dim)(x)\n    x = tf.reshape(x, (-,,,dim))\n    x = tf.transpose(x,(,,,))\n    x = L.BatchNormalization()(x)\n    x = L.Activation()(x)\n</code></pre>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275433\" target=\"_blank\">8th Place Solution</a><ul>\n<li>Data was multiplied by 1e19 (1e21 later) to ensure that CQT and CWT could be performed correctly using FP16.</li>\n<li>Input images: used nnAudio CQT (or CWT) as the first model layer. Then 3 outputs are stacked on frequency axis. Then divided the image by NEG (average of negative sample images) to get rid of noise on average. Finally applied a log scaling. Depending on the model added the mask as an additional channel. Then scaled data to have multiple of 16 dimensions.</li>\n<li>Mostly used efficientnet_v2s_s and efficientnet_v2_m from timm package.</li>\n<li>Used BCEWithLogitsLoss, Adam and OneCycleLR.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275353\" target=\"_blank\">10th Place Solution</a><ul>\n<li>CWT + Resnet34/EfNet, 488x512, 1/5 fold, train 5 epochs</li>\n<li>Replace CWT with a multiple layers 1D Conv, the same size as CWT, train 20 epochs</li>\n<li>Using different parameter sets(bandpass, time shift, random channel off etc.) to train 10 models above. Ensemble them all.</li>\n<li>Augmentation<ul>\n<li>Trim the original wave to (3, 3904), Random shift +- 65</li>\n<li>Random shift each channel +- 5</li>\n<li>Random turn off one channel</li>\n<li>Random switch channel of two waves (target=0 only)</li></ul></li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275405\" target=\"_blank\">11th Place Solution</a><ul>\n<li>preprocessing: bpf 30-500, bpf 25-1020</li>\n<li>augmentation: mixup</li>\n<li>1DCNN models: used residual connection and dilated convolution/standard convolution with ~24 layers, ~2M parameters.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275356\" target=\"_blank\">12nd Place Solution</a><ul>\n<li>Divide the all wave form by 4.6152116213830774e-20 (max(abs) of the entire train and test data).</li>\n<li>Use nnAudio to run CQT. Randomly select one of flattop, blackmanharris, or nuttall to the window.</li>\n<li>The spectrograms are combined in the frequency direction and input to the model as a single-channel image</li>\n<li>Resize to 384x512 before inputting into the model, and normalize the spectrogram of the entire data set in mean, std</li>\n<li>Augmentation: Before converting to a spectrogram, mixup. Randomly roll shift in time direction</li>\n<li>model: timm tf_efficientnet_b4_ap, Adam, CosineAnnealingLR</li>\n<li>Just one part of the final solution. Lots of models in the final stacking!</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275841\" target=\"_blank\">13th Place Solution</a><ul>\n<li>3 Channel (native) - stacked the detectors channel wise.</li>\n<li>6 Permutations - Horizontally stacked the 3 detectors with different permutations. Then stacked these 6 permutations channel wise.</li>\n<li>CQT+CWT - Took 3 channels(for 3 detectors) from CQT transformation and 3 from CWT transformation and stacked them.</li>\n<li>Double CWT - created two parallel CWT transformations with different parameters</li>\n<li>Stacking</li></ul></li>\n<li><a href=\"https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275431\" target=\"_blank\">15th Place Solution</a><ul>\n<li>Conv2D</li>\n<li>Whitening: Using average PSD. Averaging over all noise samples for each site.</li>\n<li>CQT Scaling with filter_scale = 8/bins_per_octave and (fmin, fmax)=(20, 1024). Both abs and angle part were used.</li>\n<li>Augmentation<ul>\n<li>Horizontal/time shift</li>\n<li>Pad both side and then horizontal random crop to get time shift image.</li>\n<li>Mixup, prevent from overfitting</li>\n<li>GeM Fixed power 3 was better than the trainable case.</li></ul></li></ul></li>\n</ul>",
      "rawMarkdown": "- [1st Place Solution] [Part 1](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275476) [Part 2](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275507)\n    - Synthetic data generation!!!\n    - Conv1D\n    - optimizer: SGD, wd=1e-4, nesterov momentum, learning rate: 0.1 with cosine annealing\n    - input: 3 channels of raw signal filtered with butterworth filter at 20hz\n    - augmentations: freq masking, time masking, channel shuffle (only Hanford, Livingston), minor time shifts between channels, 5ms\n- [2nd Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275341)\n    - Trainable frontend in general outperformed fixed frontend.\n    - 1D CNN, WaveNet wavegram, CWT.\n    - Applied bandpass filter to all networks. [16, 512] for CWT-CNN and Trainable frontend CNN, [30, 300] for 1d-CNN. Whitening did not work.\n    - Augmentations: gaussian noise for 2d-CNN networks, and flipped wave amplitude for 1d-CNN.\n    - Re-training on soft(continuous) pseudo-labelled test dataset improved AUC by ~0.001. Label smoothing during pseudo-label also helped a bit.\n    - Cross validated Ridge regression model was used to combine the outputs from neural network models.\n- [3rd Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275617)\n    - Pretraining with GW, Training with Pseudo Label or Soft Label, BCE, Rank Loss, AdamW, RangerLars+Lookahead optimizer\n    - Preprocessing:\n        - Avg PSD of target 0 (Design Curves)\n        - Extending waves\n        - Whitening with Tukey window\n    - 2D models:\n        - EfficientNet(B3, B4, B5, B7), EfficientNetV2(M), ResNet200D, Inception-V3 (also we performed a number of initial experiments with ResNeXt models)\n        - CQT and CWT images generated based on the whitened signal\n        - Image size 128 x 128 〜 512 x 512\n        - Soft leak-free pseudo labeling from ensemble results\n        - LIGO channel swap argumentation (randomly swapping LIGO channels) for both training and TTA\n        - 1D mixup prior to CQT/CWT\n        - Adding a 4th channel to the spectrogram/scalogram input which is just a linear gradient (-1, 1) along the frequency dimension used for frequency encoding (similar to positional encoding in transformers)\n- [4th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275331)\n    - GW waves numpy array --> horizontal stacking all three to get (1,4096*3) array --> band pass filtering ---> Deep conv1d backbone with residuals --->LSTM head ---> Prediction\n    - Numpy ---> signal tukey ---> band pass filter ---> normalized by norm_by =[7.729773e-21,8.228142e-21, 8.750003e-21] --->CQT--> Augmentations[coloredNoise and shift]\n    - Colored noise augmentation was done channel wise while shift was applied sample wise.\n- [5th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275334)\n    - PSD whitening with precomputed mean PSD for each channel, custom window function : mirrored sigmoid\n    - 1d augs: shift, noise, mixup, cutmix, shuffle\n    - 2d CQT default settings fmin=16, fmax=1024, hop=12\n    - 5xCNN 2d ensemble with TTA, models and EMA's\n- [6th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275343)\n    - wave -> dct -> trainable bp filter -> idct -> 1dcnn/cwt -> 1dcnn/2dcnn/resnet/effnetv2/lstm\n    - best single model：4096x3 -> 1dcnn -> 512x256x3 -> resnet34\n    - augmentation: random shift wave separately, up to 1/32 second, random change phase\n- [7th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275349)\n    - Efficient B3 (size=128*128), Wavenet + GRU, Wavenet\n    - Preprocess: 30Hz highpass filtering, normalized by the absolute means of individual observations\n    - Augmentation: Roll +/-500 points (p=0.5), Scale 0.85-1.15 (p=0.5), Multiply -1 (p=0.5)\n    \n```python\n    # The bellow frontend is used by all models. dim=128\n    input = L.Input(shape=(3, 4096)) # 3 observations\n    x = tf.reshape(input,(-1,4096,1)) # 3 observations folded into batch\n    x = wavenet(x, dim, dilations=12, kernel_size=5)\n    x = L.Dense(dim//4)(x)\n    x = wavenet(x, dim, dilations=12, kernel_size=5)\n    x = L.Dense(dim//4)(x)\n    x = L.Dense(dim)(x)\n    x = tf.reshape(x, (-1,3,4096,dim))\n    x = tf.transpose(x,(0,2,3,1))\n    x = L.BatchNormalization()(x)\n    x = L.Activation('gelu')(x)\n```\n- [8th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275433)\n    - Data was multiplied by 1e19 (1e21 later) to ensure that CQT and CWT could be performed correctly using FP16.\n    - Input images: used nnAudio CQT (or CWT) as the first model layer. Then 3 outputs are stacked on frequency axis. Then divided the image by NEG (average of negative sample images) to get rid of noise on average. Finally applied a log scaling. Depending on the model added the mask as an additional channel. Then scaled data to have multiple of 16 dimensions.\n    - Mostly used efficientnet_v2s_s and efficientnet_v2_m from timm package.\n    - Used BCEWithLogitsLoss, Adam and OneCycleLR.\n- [10th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275353)\n    - CWT + Resnet34/EfNet, 488x512, 1/5 fold, train 5 epochs\n    - Replace CWT with a multiple layers 1D Conv, the same size as CWT, train 20 epochs\n    - Using different parameter sets(bandpass, time shift, random channel off etc.) to train 10 models above. Ensemble them all.\n    - Augmentation\n        - Trim the original wave to (3, 3904), Random shift +- 65\n        - Random shift each channel +- 5\n        - Random turn off one channel\n        - Random switch channel of two waves (target=0 only)\n- [11th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275405)\n    - preprocessing: bpf 30-500, bpf 25-1020\n    - augmentation: mixup\n    - 1DCNN models: used residual connection and dilated convolution/standard convolution with ~24 layers, ~2M parameters.\n- [12nd Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275356)\n    - Divide the all wave form by 4.6152116213830774e-20 (max(abs) of the entire train and test data).\n    - Use nnAudio to run CQT. Randomly select one of flattop, blackmanharris, or nuttall to the window.\n    - The spectrograms are combined in the frequency direction and input to the model as a single-channel image\n    - Resize to 384x512 before inputting into the model, and normalize the spectrogram of the entire data set in mean, std\n    - Augmentation: Before converting to a spectrogram, mixup. Randomly roll shift in time direction\n    - model: timm tf_efficientnet_b4_ap, Adam, CosineAnnealingLR\n    - Just one part of the final solution. Lots of models in the final stacking!\n- [13th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275841)\n    - 3 Channel (native) - stacked the detectors channel wise.\n    - 6 Permutations - Horizontally stacked the 3 detectors with different permutations. Then stacked these 6 permutations channel wise.\n    - CQT+CWT - Took 3 channels(for 3 detectors) from CQT transformation and 3 from CWT transformation and stacked them.\n    - Double CWT - created two parallel CWT transformations with different parameters\n    - Stacking\n- [15th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275431)\n    - Conv2D\n    - Whitening: Using average PSD. Averaging over all noise samples for each site.\n    - CQT Scaling with filter_scale = 8/bins_per_octave and (fmin, fmax)=(20, 1024). Both abs and angle part were used.\n    - Augmentation\n        - Horizontal/time shift\n        - Pad both side and then horizontal random crop to get time shift image.\n        - Mixup, prevent from overfitting\n        - GeM Fixed power 3 was better than the trainable case.",
      "votes": 34
    },
    {
      "id": 2012674,
      "postDate": "2022-11-01T10:44:48.380Z",
      "content": "<p>The previous competition was far different from this one.</p>\n<p>[1st] Their data had a much higher number of training instances<br>\n[2nd] Their data was in time domain<br>\n[3rd] they had a much smaller dimensional size (input shape: 3x4096) // we have instead 2x360x2x4096</p>\n<p>This means that this competition will possible need a total different approach.</p>",
      "rawMarkdown": "The previous competition was far different from this one.\n\n[1st] Their data had a much higher number of training instances\n[2nd] Their data was in time domain\n[3rd] they had a much smaller dimensional size (input shape: 3x4096) // we have instead 2x360x2x4096\n\nThis means that this competition will possible need a total different approach.",
      "votes": 8,
      "replies": [
        {
          "id": 2016179,
          "postDate": "2022-11-03T20:30:39.153Z",
          "content": "<p>Nice! I was looking forward to this only. Thanks 🙏</p>",
          "rawMarkdown": "Nice! I was looking forward to this only. Thanks 🙏"
        }
      ]
    },
    {
      "id": 2016181,
      "postDate": "2022-11-03T20:31:47.307Z",
      "content": "<p>Thanks for sharing this..will check and see if this can be relevant here. I am new to domain so not sure about it though. </p>",
      "rawMarkdown": "Thanks for sharing this..will check and see if this can be relevant here. I am new to domain so not sure about it though. ",
      "votes": 1
    },
    {
      "id": 2014082,
      "postDate": "2022-11-02T09:52:51.457Z",
      "content": "<p>Thank you for this information:) The first's place model looks simple. The synthetic data generation had to be really powerfull</p>",
      "rawMarkdown": "Thank you for this information:) The first's place model looks simple. The synthetic data generation had to be really powerfull",
      "votes": 1
    },
    {
      "id": 2011895,
      "postDate": "2022-10-31T23:22:08.807Z",
      "content": "<p>Thanks for spending the time to compile all this info!</p>",
      "rawMarkdown": "Thanks for spending the time to compile all this info!",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 2012674,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-11-01T10:44:48.380000",
      "content": "<p>The previous competition was far different from this one.</p>\n<p>[1st] Their data had a much higher number of training instances<br>\n[2nd] Their data was in time domain<br>\n[3rd] they had a much smaller dimensional size (input shape: 3x4096) // we have instead 2x360x2x4096</p>\n<p>This means that this competition will possible need a total different approach.</p>",
      "votes": 8,
      "replies": [
        {
          "id": 2016179,
          "author_name": "Chirag Desai",
          "author_url": "",
          "post_date": "2022-11-03T20:30:39.153000",
          "content": "<p>Nice! I was looking forward to this only. Thanks 🙏</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2016181,
      "author_name": "Chirag Desai",
      "author_url": "",
      "post_date": "2022-11-03T20:31:47.307000",
      "content": "<p>Thanks for sharing this..will check and see if this can be relevant here. I am new to domain so not sure about it though. </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2014082,
      "author_name": "Nunyapa",
      "author_url": "",
      "post_date": "2022-11-02T09:52:51.457000",
      "content": "<p>Thank you for this information:) The first's place model looks simple. The synthetic data generation had to be really powerfull</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2011895,
      "author_name": "aspiring",
      "author_url": "",
      "post_date": "2022-10-31T23:22:08.807000",
      "content": "<p>Thanks for spending the time to compile all this info!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2011892": "- [1st Place Solution] [Part 1](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275476) [Part 2](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275507)\n    - Synthetic data generation!!!\n    - Conv1D\n    - optimizer: SGD, wd=1e-4, nesterov momentum, learning rate: 0.1 with cosine annealing\n    - input: 3 channels of raw signal filtered with butterworth filter at 20hz\n    - augmentations: freq masking, time masking, channel shuffle (only Hanford, Livingston), minor time shifts between channels, 5ms\n- [2nd Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275341)\n    - Trainable frontend in general outperformed fixed frontend.\n    - 1D CNN, WaveNet wavegram, CWT.\n    - Applied bandpass filter to all networks. [16, 512] for CWT-CNN and Trainable frontend CNN, [30, 300] for 1d-CNN. Whitening did not work.\n    - Augmentations: gaussian noise for 2d-CNN networks, and flipped wave amplitude for 1d-CNN.\n    - Re-training on soft(continuous) pseudo-labelled test dataset improved AUC by ~0.001. Label smoothing during pseudo-label also helped a bit.\n    - Cross validated Ridge regression model was used to combine the outputs from neural network models.\n- [3rd Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275617)\n    - Pretraining with GW, Training with Pseudo Label or Soft Label, BCE, Rank Loss, AdamW, RangerLars+Lookahead optimizer\n    - Preprocessing:\n        - Avg PSD of target 0 (Design Curves)\n        - Extending waves\n        - Whitening with Tukey window\n    - 2D models:\n        - EfficientNet(B3, B4, B5, B7), EfficientNetV2(M), ResNet200D, Inception-V3 (also we performed a number of initial experiments with ResNeXt models)\n        - CQT and CWT images generated based on the whitened signal\n        - Image size 128 x 128 〜 512 x 512\n        - Soft leak-free pseudo labeling from ensemble results\n        - LIGO channel swap argumentation (randomly swapping LIGO channels) for both training and TTA\n        - 1D mixup prior to CQT/CWT\n        - Adding a 4th channel to the spectrogram/scalogram input which is just a linear gradient (-1, 1) along the frequency dimension used for frequency encoding (similar to positional encoding in transformers)\n- [4th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275331)\n    - GW waves numpy array --> horizontal stacking all three to get (1,4096*3) array --> band pass filtering ---> Deep conv1d backbone with residuals --->LSTM head ---> Prediction\n    - Numpy ---> signal tukey ---> band pass filter ---> normalized by norm_by =[7.729773e-21,8.228142e-21, 8.750003e-21] --->CQT--> Augmentations[coloredNoise and shift]\n    - Colored noise augmentation was done channel wise while shift was applied sample wise.\n- [5th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275334)\n    - PSD whitening with precomputed mean PSD for each channel, custom window function : mirrored sigmoid\n    - 1d augs: shift, noise, mixup, cutmix, shuffle\n    - 2d CQT default settings fmin=16, fmax=1024, hop=12\n    - 5xCNN 2d ensemble with TTA, models and EMA's\n- [6th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275343)\n    - wave -> dct -> trainable bp filter -> idct -> 1dcnn/cwt -> 1dcnn/2dcnn/resnet/effnetv2/lstm\n    - best single model：4096x3 -> 1dcnn -> 512x256x3 -> resnet34\n    - augmentation: random shift wave separately, up to 1/32 second, random change phase\n- [7th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275349)\n    - Efficient B3 (size=128*128), Wavenet + GRU, Wavenet\n    - Preprocess: 30Hz highpass filtering, normalized by the absolute means of individual observations\n    - Augmentation: Roll +/-500 points (p=0.5), Scale 0.85-1.15 (p=0.5), Multiply -1 (p=0.5)\n    \n```python\n    # The bellow frontend is used by all models. dim=128\n    input = L.Input(shape=(3, 4096)) # 3 observations\n    x = tf.reshape(input,(-1,4096,1)) # 3 observations folded into batch\n    x = wavenet(x, dim, dilations=12, kernel_size=5)\n    x = L.Dense(dim//4)(x)\n    x = wavenet(x, dim, dilations=12, kernel_size=5)\n    x = L.Dense(dim//4)(x)\n    x = L.Dense(dim)(x)\n    x = tf.reshape(x, (-1,3,4096,dim))\n    x = tf.transpose(x,(0,2,3,1))\n    x = L.BatchNormalization()(x)\n    x = L.Activation('gelu')(x)\n```\n- [8th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275433)\n    - Data was multiplied by 1e19 (1e21 later) to ensure that CQT and CWT could be performed correctly using FP16.\n    - Input images: used nnAudio CQT (or CWT) as the first model layer. Then 3 outputs are stacked on frequency axis. Then divided the image by NEG (average of negative sample images) to get rid of noise on average. Finally applied a log scaling. Depending on the model added the mask as an additional channel. Then scaled data to have multiple of 16 dimensions.\n    - Mostly used efficientnet_v2s_s and efficientnet_v2_m from timm package.\n    - Used BCEWithLogitsLoss, Adam and OneCycleLR.\n- [10th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275353)\n    - CWT + Resnet34/EfNet, 488x512, 1/5 fold, train 5 epochs\n    - Replace CWT with a multiple layers 1D Conv, the same size as CWT, train 20 epochs\n    - Using different parameter sets(bandpass, time shift, random channel off etc.) to train 10 models above. Ensemble them all.\n    - Augmentation\n        - Trim the original wave to (3, 3904), Random shift +- 65\n        - Random shift each channel +- 5\n        - Random turn off one channel\n        - Random switch channel of two waves (target=0 only)\n- [11th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275405)\n    - preprocessing: bpf 30-500, bpf 25-1020\n    - augmentation: mixup\n    - 1DCNN models: used residual connection and dilated convolution/standard convolution with ~24 layers, ~2M parameters.\n- [12nd Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275356)\n    - Divide the all wave form by 4.6152116213830774e-20 (max(abs) of the entire train and test data).\n    - Use nnAudio to run CQT. Randomly select one of flattop, blackmanharris, or nuttall to the window.\n    - The spectrograms are combined in the frequency direction and input to the model as a single-channel image\n    - Resize to 384x512 before inputting into the model, and normalize the spectrogram of the entire data set in mean, std\n    - Augmentation: Before converting to a spectrogram, mixup. Randomly roll shift in time direction\n    - model: timm tf_efficientnet_b4_ap, Adam, CosineAnnealingLR\n    - Just one part of the final solution. Lots of models in the final stacking!\n- [13th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275841)\n    - 3 Channel (native) - stacked the detectors channel wise.\n    - 6 Permutations - Horizontally stacked the 3 detectors with different permutations. Then stacked these 6 permutations channel wise.\n    - CQT+CWT - Took 3 channels(for 3 detectors) from CQT transformation and 3 from CWT transformation and stacked them.\n    - Double CWT - created two parallel CWT transformations with different parameters\n    - Stacking\n- [15th Place Solution](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275431)\n    - Conv2D\n    - Whitening: Using average PSD. Averaging over all noise samples for each site.\n    - CQT Scaling with filter_scale = 8/bins_per_octave and (fmin, fmax)=(20, 1024). Both abs and angle part were used.\n    - Augmentation\n        - Horizontal/time shift\n        - Pad both side and then horizontal random crop to get time shift image.\n        - Mixup, prevent from overfitting\n        - GeM Fixed power 3 was better than the trainable case.",
    "2012674": "The previous competition was far different from this one.\n\n[1st] Their data had a much higher number of training instances\n[2nd] Their data was in time domain\n[3rd] they had a much smaller dimensional size (input shape: 3x4096) // we have instead 2x360x2x4096\n\nThis means that this competition will possible need a total different approach.",
    "2016181": "Thanks for sharing this..will check and see if this can be relevant here. I am new to domain so not sure about it though. ",
    "2014082": "Thank you for this information:) The first's place model looks simple. The synthetic data generation had to be really powerfull",
    "2011895": "Thanks for spending the time to compile all this info!"
  }
}