{
  "id": 375927,
  "title": "22th-place solution : simulated CW signals & augmentations",
  "url": "/competitions/g2net-detecting-continuous-gravitational-waves/discussion/375927",
  "author_name": "HyeongChan Kim",
  "post_date": "2023-01-04T03:39:16.387000",
  "votes": 12,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Hello everyone!</p>\n<p>First, thanks to EGO for hosting an exciting competition! Also, congratulations to all the winners!</p>\n<h2>Data</h2>\n<h3>Pre-Processing</h3>\n<p>In my experiment, <a href=\"https://www.kaggle.com/code/laeyoung/g2net-large-kernel-inference\" target=\"_blank\">preprocessing</a> (<code>normalize</code> function) works better than the power spectrogram. It improves the score by about +0.02 on CV/LB. After normalizing the signal, take a mean over the time axis. The final shape is (360, 360).</p>\n<h3>Simulation</h3>\n<p>Generating samples is the most crucial part of boosting the score. I can get 0.761 on the LB with a single model.</p>\n<p>In short, signal depth (<code>sqrtSX / h0</code>) takes a huge impact. I generated 100K samples (50K positives, 50K negatives) and uniformly sampled the signal depth between 10 and 100. <code>cosi</code> parameter is uniformly sampled (-1, 1).</p>\n<table>\n<thead>\n<tr>\n<th>signal depth</th>\n<th>LB score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>10 ~ 50</td>\n<td>0.73x ~ 0.74x</td>\n</tr>\n<tr>\n<td>10 ~ 80</td>\n<td>0.75x</td>\n</tr>\n<tr>\n<td>10 ~ 100</td>\n<td>0.761</td>\n</tr>\n</tbody>\n</table>\n<h3>Augmentations</h3>\n<p>Also, I've worked on the augmentations for much time. Here's a list.</p>\n<ol>\n<li>v/hflip</li>\n<li>shuffle channel</li>\n<li>shift on freq-axis </li>\n<li>denoise a signal (subtract corresponding noise from the signal)</li>\n<li>add noises<ul>\n<li>Guassian N(0, 1e-2)</li>\n<li>mixed (add or concatenate) with another (stationary) noise(s)</li></ul></li>\n<li>add vertical line artifact(s).</li>\n<li>SpecAugment</li>\n<li>mixup (alpha 5.0)<ul>\n<li>perform <code>or</code> mixup</li></ul></li>\n</ol>\n<h2>Model</h2>\n<p>First, I tried to search for the backbones (effnet, nfnet, resnest, convnext, vit-based) and found <code>convnext</code> works best on CV &amp; LB score. After selecting a baseline backbone, I experimented with customizing a stem layer (e.g. large kernel &amp; pool sizes, multiple convolutions stem with various kernel sizes) to detect the long-lasting signal effectively, but they didn't affect the performance positively.</p>\n<h2>Ensemble</h2>\n<p>Most of the models used at the ensemble are <code>convnext-xlarge</code> but each model trained with different variances (e.g. augmentations, simulated samples, …) and <code>eca-nfnet-l2</code>, <code>efficientnetv2-xl</code> for one model. Every model trained on various datasets and LB score seems reliable, so I adjusted the ensemble weights by LB score.</p>\n<p>I selected the two best LB submissions (LB 0.768 PB 0.771). And the best PB that I didn't select is 0.778 (LB 0.766) (mixing all my experiments).</p>\n<h2>Works</h2>\n<ul>\n<li><code>convnext</code> family backbone</li>\n<li>signal depth 10 ~ 100</li>\n<li>hard augmentation</li>\n<li>pair stratified k fold<ul>\n<li>8 folds</li>\n<li>stratified on the target</li>\n<li><code>pair</code> means the pair (corresponding noise &amp; signal) must be in the same fold.</li></ul></li>\n<li>pseudo label (smooth label)</li>\n<li>segmentation (but hard to converge on my experiment)</li>\n<li>TTA</li>\n</ul>\n<h2>Not Works</h2>\n<ul>\n<li>segmentation with classification head (0.6 * bce + 0.4 * dice)<ul>\n<li>Actually, seg with cls works slightly better than only cls, but hard to train without loss divergence. So, I just did only cls.</li></ul></li>\n<li><code>cosi == 0</code><ul>\n<li><code>cosi</code> is also a critical parameter to determine an SNR. I generated more samples where <code>cosi</code> is 0, but there's a score drop.</li></ul></li>\n<li>augmentations (not worked)<ul>\n<li>swap with random negatives (proposed at the past competition)</li>\n<li>random resized crop</li></ul></li>\n<li>Customize a stem layer with large kernel &amp; pool sizes.</li>\n</ul>\n<p>I hope this could help you :)</p>\n<p>Happy new year!</p>",
  "messages": [
    {
      "id": 2085289,
      "postDate": "2023-01-04T03:39:16.387Z",
      "content": "<p>Hello everyone!</p>\n<p>First, thanks to EGO for hosting an exciting competition! Also, congratulations to all the winners!</p>\n<h2>Data</h2>\n<h3>Pre-Processing</h3>\n<p>In my experiment, <a href=\"https://www.kaggle.com/code/laeyoung/g2net-large-kernel-inference\" target=\"_blank\">preprocessing</a> (<code>normalize</code> function) works better than the power spectrogram. It improves the score by about +0.02 on CV/LB. After normalizing the signal, take a mean over the time axis. The final shape is (360, 360).</p>\n<h3>Simulation</h3>\n<p>Generating samples is the most crucial part of boosting the score. I can get 0.761 on the LB with a single model.</p>\n<p>In short, signal depth (<code>sqrtSX / h0</code>) takes a huge impact. I generated 100K samples (50K positives, 50K negatives) and uniformly sampled the signal depth between 10 and 100. <code>cosi</code> parameter is uniformly sampled (-1, 1).</p>\n<table>\n<thead>\n<tr>\n<th>signal depth</th>\n<th>LB score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>10 ~ 50</td>\n<td>0.73x ~ 0.74x</td>\n</tr>\n<tr>\n<td>10 ~ 80</td>\n<td>0.75x</td>\n</tr>\n<tr>\n<td>10 ~ 100</td>\n<td>0.761</td>\n</tr>\n</tbody>\n</table>\n<h3>Augmentations</h3>\n<p>Also, I've worked on the augmentations for much time. Here's a list.</p>\n<ol>\n<li>v/hflip</li>\n<li>shuffle channel</li>\n<li>shift on freq-axis </li>\n<li>denoise a signal (subtract corresponding noise from the signal)</li>\n<li>add noises<ul>\n<li>Guassian N(0, 1e-2)</li>\n<li>mixed (add or concatenate) with another (stationary) noise(s)</li></ul></li>\n<li>add vertical line artifact(s).</li>\n<li>SpecAugment</li>\n<li>mixup (alpha 5.0)<ul>\n<li>perform <code>or</code> mixup</li></ul></li>\n</ol>\n<h2>Model</h2>\n<p>First, I tried to search for the backbones (effnet, nfnet, resnest, convnext, vit-based) and found <code>convnext</code> works best on CV &amp; LB score. After selecting a baseline backbone, I experimented with customizing a stem layer (e.g. large kernel &amp; pool sizes, multiple convolutions stem with various kernel sizes) to detect the long-lasting signal effectively, but they didn't affect the performance positively.</p>\n<h2>Ensemble</h2>\n<p>Most of the models used at the ensemble are <code>convnext-xlarge</code> but each model trained with different variances (e.g. augmentations, simulated samples, …) and <code>eca-nfnet-l2</code>, <code>efficientnetv2-xl</code> for one model. Every model trained on various datasets and LB score seems reliable, so I adjusted the ensemble weights by LB score.</p>\n<p>I selected the two best LB submissions (LB 0.768 PB 0.771). And the best PB that I didn't select is 0.778 (LB 0.766) (mixing all my experiments).</p>\n<h2>Works</h2>\n<ul>\n<li><code>convnext</code> family backbone</li>\n<li>signal depth 10 ~ 100</li>\n<li>hard augmentation</li>\n<li>pair stratified k fold<ul>\n<li>8 folds</li>\n<li>stratified on the target</li>\n<li><code>pair</code> means the pair (corresponding noise &amp; signal) must be in the same fold.</li></ul></li>\n<li>pseudo label (smooth label)</li>\n<li>segmentation (but hard to converge on my experiment)</li>\n<li>TTA</li>\n</ul>\n<h2>Not Works</h2>\n<ul>\n<li>segmentation with classification head (0.6 * bce + 0.4 * dice)<ul>\n<li>Actually, seg with cls works slightly better than only cls, but hard to train without loss divergence. So, I just did only cls.</li></ul></li>\n<li><code>cosi == 0</code><ul>\n<li><code>cosi</code> is also a critical parameter to determine an SNR. I generated more samples where <code>cosi</code> is 0, but there's a score drop.</li></ul></li>\n<li>augmentations (not worked)<ul>\n<li>swap with random negatives (proposed at the past competition)</li>\n<li>random resized crop</li></ul></li>\n<li>Customize a stem layer with large kernel &amp; pool sizes.</li>\n</ul>\n<p>I hope this could help you :)</p>\n<p>Happy new year!</p>",
      "rawMarkdown": "Hello everyone!\n\nFirst, thanks to EGO for hosting an exciting competition! Also, congratulations to all the winners!\n\n## Data\n\n### Pre-Processing\n\nIn my experiment, [preprocessing](https://www.kaggle.com/code/laeyoung/g2net-large-kernel-inference) (`normalize` function) works better than the power spectrogram. It improves the score by about +0.02 on CV/LB. After normalizing the signal, take a mean over the time axis. The final shape is (360, 360).\n\n### Simulation\n\nGenerating samples is the most crucial part of boosting the score. I can get 0.761 on the LB with a single model.\n\nIn short, signal depth (`sqrtSX / h0`) takes a huge impact. I generated 100K samples (50K positives, 50K negatives) and uniformly sampled the signal depth between 10 and 100. `cosi` parameter is uniformly sampled (-1, 1).\n\n| signal depth  | LB score  |\n| --- | --- |\n| 10 ~ 50 | 0.73x ~ 0.74x |\n| 10 ~ 80 | 0.75x |\n| 10 ~ 100 | 0.761 |\n\n### Augmentations\n\nAlso, I've worked on the augmentations for much time. Here's a list.\n\n1. v/hflip\n2. shuffle channel\n3. shift on freq-axis \n4. denoise a signal (subtract corresponding noise from the signal)\n5. add noises\n  * Guassian N(0, 1e-2)\n  * mixed (add or concatenate) with another (stationary) noise(s)\n6. add vertical line artifact(s).\n7. SpecAugment\n8. mixup (alpha 5.0)\n  * perform `or` mixup\n\n## Model\n\nFirst, I tried to search for the backbones (effnet, nfnet, resnest, convnext, vit-based) and found `convnext` works best on CV & LB score. After selecting a baseline backbone, I experimented with customizing a stem layer (e.g. large kernel & pool sizes, multiple convolutions stem with various kernel sizes) to detect the long-lasting signal effectively, but they didn't affect the performance positively.\n\n## Ensemble\n\nMost of the models used at the ensemble are `convnext-xlarge` but each model trained with different variances (e.g. augmentations, simulated samples, ...) and `eca-nfnet-l2`, `efficientnetv2-xl` for one model. Every model trained on various datasets and LB score seems reliable, so I adjusted the ensemble weights by LB score.\n\nI selected the two best LB submissions (LB 0.768 PB 0.771). And the best PB that I didn't select is 0.778 (LB 0.766) (mixing all my experiments).\n\n## Works\n\n* `convnext` family backbone\n* signal depth 10 ~ 100\n* hard augmentation\n* pair stratified k fold\n  * 8 folds\n  * stratified on the target\n  * `pair` means the pair (corresponding noise & signal) must be in the same fold.\n* pseudo label (smooth label)\n* segmentation (but hard to converge on my experiment)\n* TTA\n\n## Not Works\n\n* segmentation with classification head (0.6 * bce + 0.4 * dice)\n  * Actually, seg with cls works slightly better than only cls, but hard to train without loss divergence. So, I just did only cls.\n* `cosi == 0`\n  * `cosi` is also a critical parameter to determine an SNR. I generated more samples where `cosi` is 0, but there's a score drop.\n* augmentations (not worked)\n  * swap with random negatives (proposed at the past competition)\n  * random resized crop\n* Customize a stem layer with large kernel & pool sizes.\n\nI hope this could help you :)\n\nHappy new year!",
      "votes": 12
    },
    {
      "id": 2091297,
      "postDate": "2023-01-08T08:17:58.960Z",
      "content": "<p>To share one observation, I think that at cosi=0 mixing noise with a low snr, [3, 10] would work better.</p>",
      "rawMarkdown": "To share one observation, I think that at cosi=0 mixing noise with a low snr, [3, 10] would work better.",
      "votes": 1,
      "replies": [
        {
          "id": 2091445,
          "postDate": "2023-01-08T12:00:52.640Z",
          "content": "<p>I second this! As you said, I also found that the model can't learn anything from high SNR with cosi=0 cases. I should have tried that.<br>\nThanks for sharing :)</p>",
          "rawMarkdown": "I second this! As you said, I also found that the model can't learn anything from high SNR with cosi=0 cases. I should have tried that.\nThanks for sharing :)"
        }
      ]
    },
    {
      "id": 2086353,
      "postDate": "2023-01-04T18:10:51.367Z",
      "content": "<p>Awesome solution! Congrats <a href=\"https://www.kaggle.com/kozistr\" target=\"_blank\">@kozistr</a>!</p>",
      "rawMarkdown": "Awesome solution! Congrats @kozistr!",
      "votes": 1
    },
    {
      "id": 2085544,
      "postDate": "2023-01-04T08:15:00.437Z",
      "content": "<p>Thanks.  I should have tried convnext and more augmentations perhaps instead of focusing on cleaning the test data.</p>",
      "rawMarkdown": "Thanks.  I should have tried convnext and more augmentations perhaps instead of focusing on cleaning the test data.",
      "votes": 1
    },
    {
      "id": 2085483,
      "postDate": "2023-01-04T07:33:00.573Z",
      "content": "<p>Congrats HyeongChan Kim！Very excellent solution! </p>",
      "rawMarkdown": "Congrats HyeongChan Kim！Very excellent solution! ",
      "votes": 1
    },
    {
      "id": 2085447,
      "postDate": "2023-01-04T07:04:14.290Z",
      "content": "<p>Very cool analysis of the importance of depth. I stopped short of going below 50. Damn.</p>",
      "rawMarkdown": "Very cool analysis of the importance of depth. I stopped short of going below 50. Damn.",
      "votes": 1
    },
    {
      "id": 2085316,
      "postDate": "2023-01-04T04:10:36.060Z",
      "content": "<p>Thanks for sharing your deep learning solution and simulation configuration! I was confused with my depth choice because I thought large depth are necessary but my experiments seem to show small depth ~ 20 is sufficient. Great to know that large depth are indeed important!</p>",
      "rawMarkdown": "Thanks for sharing your deep learning solution and simulation configuration! I was confused with my depth choice because I thought large depth are necessary but my experiments seem to show small depth ~ 20 is sufficient. Great to know that large depth are indeed important!",
      "votes": 1,
      "replies": [
        {
          "id": 2085329,
          "postDate": "2023-01-04T04:26:24.697Z",
          "content": "<p>First of all, huge congrats to win the competition! I learned from your solution a lot : )</p>\n<p>I also experimented with different depth ranges (10 ~ 50, 10 ~ 80, 10 ~ 100), and there's a performance gap</p>\n<ul>\n<li>10 ~ 50 : LB 0.73x ~ 0.74x</li>\n<li>10 ~ 80 : LB 0.75x</li>\n<li>10 ~ 100 : LB 0.76x</li>\n</ul>\n<p>I also thought that a small depth is enough and the model barely learned large-depth cases. However, after checking the AUC score by depth &amp; GradCam, there's something that model catches and I guess train &amp; test sets have large-depth cases.</p>\n<p>Here's the score by the depth! (maybe slightly different, because just from my memory)</p>\n<table>\n<thead>\n<tr>\n<th>depth</th>\n<th>score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>10 ~ 20</td>\n<td>1.0</td>\n</tr>\n<tr>\n<td>20 ~ 30</td>\n<td>0.95</td>\n</tr>\n<tr>\n<td>30 ~ 40</td>\n<td>0.9</td>\n</tr>\n<tr>\n<td>40 ~ 50</td>\n<td>0.8</td>\n</tr>\n<tr>\n<td>50 ~ 60</td>\n<td>0.72</td>\n</tr>\n<tr>\n<td>60 ~ 70</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>70 ~ 80</td>\n<td>0.55</td>\n</tr>\n<tr>\n<td>80 ~ 90</td>\n<td>0.53</td>\n</tr>\n<tr>\n<td>90 ~ 100</td>\n<td>0.52</td>\n</tr>\n</tbody>\n</table>",
          "rawMarkdown": "First of all, huge congrats to win the competition! I learned from your solution a lot : )\n\nI also experimented with different depth ranges (10 ~ 50, 10 ~ 80, 10 ~ 100), and there's a performance gap\n* 10 ~ 50 : LB 0.73x ~ 0.74x\n* 10 ~ 80 : LB 0.75x\n* 10 ~ 100 : LB 0.76x\n\nI also thought that a small depth is enough and the model barely learned large-depth cases. However, after checking the AUC score by depth & GradCam, there's something that model catches and I guess train & test sets have large-depth cases.\n\nHere's the score by the depth! (maybe slightly different, because just from my memory)\n\n| depth  | score  |\n| --- | --- |\n| 10 ~ 20 | 1.0  |\n| 20 ~ 30 | 0.95  |\n| 30 ~ 40 | 0.9  |\n| 40 ~ 50 | 0.8  |\n| 50 ~ 60 | 0.72 |\n| 60 ~ 70 | 0.6 |\n| 70 ~ 80 | 0.55 |\n| 80 ~ 90 | 0.53 |\n| 90 ~ 100 | 0.52  |",
          "votes": 1
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2091297,
      "author_name": "assign",
      "author_url": "",
      "post_date": "2023-01-08T08:17:58.960000",
      "content": "<p>To share one observation, I think that at cosi=0 mixing noise with a low snr, [3, 10] would work better.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2091445,
          "author_name": "HyeongChan Kim",
          "author_url": "",
          "post_date": "2023-01-08T12:00:52.640000",
          "content": "<p>I second this! As you said, I also found that the model can't learn anything from high SNR with cosi=0 cases. I should have tried that.<br>\nThanks for sharing :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2086353,
      "author_name": "Ravi Shah",
      "author_url": "",
      "post_date": "2023-01-04T18:10:51.367000",
      "content": "<p>Awesome solution! Congrats <a href=\"https://www.kaggle.com/kozistr\" target=\"_blank\">@kozistr</a>!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2085544,
      "author_name": "Jonathan McKinney",
      "author_url": "",
      "post_date": "2023-01-04T08:15:00.437000",
      "content": "<p>Thanks.  I should have tried convnext and more augmentations perhaps instead of focusing on cleaning the test data.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2085483,
      "author_name": "BarryZhou",
      "author_url": "",
      "post_date": "2023-01-04T07:33:00.573000",
      "content": "<p>Congrats HyeongChan Kim！Very excellent solution! </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2085447,
      "author_name": "DennisSakva",
      "author_url": "",
      "post_date": "2023-01-04T07:04:14.290000",
      "content": "<p>Very cool analysis of the importance of depth. I stopped short of going below 50. Damn.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2085316,
      "author_name": "🐢 Jun Koda",
      "author_url": "",
      "post_date": "2023-01-04T04:10:36.060000",
      "content": "<p>Thanks for sharing your deep learning solution and simulation configuration! I was confused with my depth choice because I thought large depth are necessary but my experiments seem to show small depth ~ 20 is sufficient. Great to know that large depth are indeed important!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2085329,
          "author_name": "HyeongChan Kim",
          "author_url": "",
          "post_date": "2023-01-04T04:26:24.697000",
          "content": "<p>First of all, huge congrats to win the competition! I learned from your solution a lot : )</p>\n<p>I also experimented with different depth ranges (10 ~ 50, 10 ~ 80, 10 ~ 100), and there's a performance gap</p>\n<ul>\n<li>10 ~ 50 : LB 0.73x ~ 0.74x</li>\n<li>10 ~ 80 : LB 0.75x</li>\n<li>10 ~ 100 : LB 0.76x</li>\n</ul>\n<p>I also thought that a small depth is enough and the model barely learned large-depth cases. However, after checking the AUC score by depth &amp; GradCam, there's something that model catches and I guess train &amp; test sets have large-depth cases.</p>\n<p>Here's the score by the depth! (maybe slightly different, because just from my memory)</p>\n<table>\n<thead>\n<tr>\n<th>depth</th>\n<th>score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>10 ~ 20</td>\n<td>1.0</td>\n</tr>\n<tr>\n<td>20 ~ 30</td>\n<td>0.95</td>\n</tr>\n<tr>\n<td>30 ~ 40</td>\n<td>0.9</td>\n</tr>\n<tr>\n<td>40 ~ 50</td>\n<td>0.8</td>\n</tr>\n<tr>\n<td>50 ~ 60</td>\n<td>0.72</td>\n</tr>\n<tr>\n<td>60 ~ 70</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>70 ~ 80</td>\n<td>0.55</td>\n</tr>\n<tr>\n<td>80 ~ 90</td>\n<td>0.53</td>\n</tr>\n<tr>\n<td>90 ~ 100</td>\n<td>0.52</td>\n</tr>\n</tbody>\n</table>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2085289": "Hello everyone!\n\nFirst, thanks to EGO for hosting an exciting competition! Also, congratulations to all the winners!\n\n## Data\n\n### Pre-Processing\n\nIn my experiment, [preprocessing](https://www.kaggle.com/code/laeyoung/g2net-large-kernel-inference) (`normalize` function) works better than the power spectrogram. It improves the score by about +0.02 on CV/LB. After normalizing the signal, take a mean over the time axis. The final shape is (360, 360).\n\n### Simulation\n\nGenerating samples is the most crucial part of boosting the score. I can get 0.761 on the LB with a single model.\n\nIn short, signal depth (`sqrtSX / h0`) takes a huge impact. I generated 100K samples (50K positives, 50K negatives) and uniformly sampled the signal depth between 10 and 100. `cosi` parameter is uniformly sampled (-1, 1).\n\n| signal depth  | LB score  |\n| --- | --- |\n| 10 ~ 50 | 0.73x ~ 0.74x |\n| 10 ~ 80 | 0.75x |\n| 10 ~ 100 | 0.761 |\n\n### Augmentations\n\nAlso, I've worked on the augmentations for much time. Here's a list.\n\n1. v/hflip\n2. shuffle channel\n3. shift on freq-axis \n4. denoise a signal (subtract corresponding noise from the signal)\n5. add noises\n  * Guassian N(0, 1e-2)\n  * mixed (add or concatenate) with another (stationary) noise(s)\n6. add vertical line artifact(s).\n7. SpecAugment\n8. mixup (alpha 5.0)\n  * perform `or` mixup\n\n## Model\n\nFirst, I tried to search for the backbones (effnet, nfnet, resnest, convnext, vit-based) and found `convnext` works best on CV & LB score. After selecting a baseline backbone, I experimented with customizing a stem layer (e.g. large kernel & pool sizes, multiple convolutions stem with various kernel sizes) to detect the long-lasting signal effectively, but they didn't affect the performance positively.\n\n## Ensemble\n\nMost of the models used at the ensemble are `convnext-xlarge` but each model trained with different variances (e.g. augmentations, simulated samples, ...) and `eca-nfnet-l2`, `efficientnetv2-xl` for one model. Every model trained on various datasets and LB score seems reliable, so I adjusted the ensemble weights by LB score.\n\nI selected the two best LB submissions (LB 0.768 PB 0.771). And the best PB that I didn't select is 0.778 (LB 0.766) (mixing all my experiments).\n\n## Works\n\n* `convnext` family backbone\n* signal depth 10 ~ 100\n* hard augmentation\n* pair stratified k fold\n  * 8 folds\n  * stratified on the target\n  * `pair` means the pair (corresponding noise & signal) must be in the same fold.\n* pseudo label (smooth label)\n* segmentation (but hard to converge on my experiment)\n* TTA\n\n## Not Works\n\n* segmentation with classification head (0.6 * bce + 0.4 * dice)\n  * Actually, seg with cls works slightly better than only cls, but hard to train without loss divergence. So, I just did only cls.\n* `cosi == 0`\n  * `cosi` is also a critical parameter to determine an SNR. I generated more samples where `cosi` is 0, but there's a score drop.\n* augmentations (not worked)\n  * swap with random negatives (proposed at the past competition)\n  * random resized crop\n* Customize a stem layer with large kernel & pool sizes.\n\nI hope this could help you :)\n\nHappy new year!",
    "2091297": "To share one observation, I think that at cosi=0 mixing noise with a low snr, [3, 10] would work better.",
    "2086353": "Awesome solution! Congrats @kozistr!",
    "2085544": "Thanks.  I should have tried convnext and more augmentations perhaps instead of focusing on cleaning the test data.",
    "2085483": "Congrats HyeongChan Kim！Very excellent solution! ",
    "2085447": "Very cool analysis of the importance of depth. I stopped short of going below 50. Damn.",
    "2085316": "Thanks for sharing your deep learning solution and simulation configuration! I was confused with my depth choice because I thought large depth are necessary but my experiments seem to show small depth ~ 20 is sufficient. Great to know that large depth are indeed important!"
  }
}