Thank you to the hosts and the organizers for having such a challenging competition!
\nMy solution is based on a CNN. I incorporated various elements, but the following three items were particularly important.
\nI used PyFstat to generate data with the same range of timestamps, frequency, and amplitudes as the test data. The signal data was generated with a signal depth (sqrtSX/h0) in the range of [1, 50]. The number of data for each is as follows.
\n| data type | \nnumber of data | \n
|---|---|
| gap noise | \n3000 | \n
| nonstationary noise | \n2983 | \n
| signal | \n5400 | \n
I created an image with shape (C,H,W) = (2, 360, 127) by performing the following processing on each of H1 and L1.
\nsklearn.preprocessing.RobustScaler.sklearn.preprocessing.RobustScaler, which is resistant to outliers.MultiOutput model predicting target and frequency.
\nnn.BCEWithLogitsLossnn.CrossEntropyLoss of 11 classesclass CustomModel(nn.Module):\n def __init__(self, pretrained=True):\n super().__init__()\n self.net = timm.create_model(\"tf_efficientnet_b5_ap\", \n pretrained=pretrained, \n num_classes=0, \n in_chans=2)\n\n # decrease first conv's stride\n modules_iter = iter(self.net.modules())\n for module in modules_iter:\n if isinstance(module, torch.nn.Conv2d) and tuple(module.stride) == (2, 2):\n break\n module.stride = (1, 2)\n\n # Target \n self.head1 = nn.Sequential(\n nn.Linear(self.net.num_features, 1)\n )\n\n # Frequency//50: 40-500Hz // 50\n freq_div_n = 50\n self.head2 = nn.Sequential(\n nn.Linear(self.net.num_features, (500 // freq_div_n) - (40 // freq_div_n) + 1)\n )\n\n def forward(self, x, labels=None):\n feat = self.net(x)\n y1 = self.head1(feat)\n y2 = self.head2(feat)\n return y1[:,0], y2\nPseudo labels using only the simulation data from the test set and pseudo labels excluding real data with amplitudes that take on outliers were effective.
\nIt was better to train with soft labels than with hard labels.
Horizontal Flip, Vertical Flip
\nRank averaging of submit.csv files from different models.
\nHowever, according to @poteman, Stacking seems to perform slightly better in private lb. (I heard about this after competition. poteman's submit is not used for final submission.)
I did all the above solutions by myself. I did not get help from my teammates.
\nCode: https://github.com/riron1206/kaggle-G2Net-Detecting-Continuous-Gravitational-Waves-10th-Place-Solution
Thank you to the hosts and the organizers for having such a challenging competition!
\nMy solution is based on a CNN. I incorporated various elements, but the following three items were particularly important.
\nI used PyFstat to generate data with the same range of timestamps, frequency, and amplitudes as the test data. The signal data was generated with a signal depth (sqrtSX/h0) in the range of [1, 50]. The number of data for each is as follows.
\n| data type | \nnumber of data | \n
|---|---|
| gap noise | \n3000 | \n
| nonstationary noise | \n2983 | \n
| signal | \n5400 | \n
I created an image with shape (C,H,W) = (2, 360, 127) by performing the following processing on each of H1 and L1.
\nsklearn.preprocessing.RobustScaler.sklearn.preprocessing.RobustScaler, which is resistant to outliers.MultiOutput model predicting target and frequency.
\nnn.BCEWithLogitsLossnn.CrossEntropyLoss of 11 classesclass CustomModel(nn.Module):\n def __init__(self, pretrained=True):\n super().__init__()\n self.net = timm.create_model(\"tf_efficientnet_b5_ap\", \n pretrained=pretrained, \n num_classes=0, \n in_chans=2)\n\n # decrease first conv's stride\n modules_iter = iter(self.net.modules())\n for module in modules_iter:\n if isinstance(module, torch.nn.Conv2d) and tuple(module.stride) == (2, 2):\n break\n module.stride = (1, 2)\n\n # Target \n self.head1 = nn.Sequential(\n nn.Linear(self.net.num_features, 1)\n )\n\n # Frequency//50: 40-500Hz // 50\n freq_div_n = 50\n self.head2 = nn.Sequential(\n nn.Linear(self.net.num_features, (500 // freq_div_n) - (40 // freq_div_n) + 1)\n )\n\n def forward(self, x, labels=None):\n feat = self.net(x)\n y1 = self.head1(feat)\n y2 = self.head2(feat)\n return y1[:,0], y2\nPseudo labels using only the simulation data from the test set and pseudo labels excluding real data with amplitudes that take on outliers were effective.
\nIt was better to train with soft labels than with hard labels.
Horizontal Flip, Vertical Flip
\nRank averaging of submit.csv files from different models.
\nHowever, according to @poteman, Stacking seems to perform slightly better in private lb. (I heard about this after competition. poteman's submit is not used for final submission.)
I did all the above solutions by myself. I did not get help from my teammates.
\nCode: https://github.com/riron1206/kaggle-G2Net-Detecting-Continuous-Gravitational-Waves-10th-Place-Solution
Cool, Congrats. I tried RobustScaler then QuantileTransformer to do the same, but both did poorly. I must have done something wrong. Did you try without frequency segmentation? How much boost was pseudo? I planned to do it, but I guessed with so much generated data based upon test data, it probably wouldn't help.
", "rawMarkdown": "Cool, Congrats. I tried RobustScaler then QuantileTransformer to do the same, but both did poorly. I must have done something wrong. Did you try without frequency segmentation? How much boost was pseudo? I planned to do it, but I guessed with so much generated data based upon test data, it probably wouldn't help.", "replies": [ { "id": 2086727, "postDate": "2023-01-05T01:17:26.657Z", "content": "Even for models that only predict targets, normalizing with RobustScaler improved public lb.
\nAfter submitting 5fold average of single model with public lb=0.747, I proceeded with the experiment by adding pseudo-labels.
\nSo I do not know the exact effect of pseudo label alone.
\nAt least, pseudo labels is essential to get public lb=0.747->0.780.
Darn :) I should have run pseudo labels.
", "rawMarkdown": "Darn :) I should have run pseudo labels." } ] } ] } ], "comments": [ { "id": 2086297, "author_name": "Jonathan McKinney", "author_url": "", "post_date": "2023-01-04T17:39:05.373000", "content": "Cool, Congrats. I tried RobustScaler then QuantileTransformer to do the same, but both did poorly. I must have done something wrong. Did you try without frequency segmentation? How much boost was pseudo? I planned to do it, but I guessed with so much generated data based upon test data, it probably wouldn't help.
", "votes": 0, "replies": [ { "id": 2086727, "author_name": "anonamename", "author_url": "", "post_date": "2023-01-05T01:17:26.657000", "content": "Even for models that only predict targets, normalizing with RobustScaler improved public lb.
\nAfter submitting 5fold average of single model with public lb=0.747, I proceeded with the experiment by adding pseudo-labels.
\nSo I do not know the exact effect of pseudo label alone.
\nAt least, pseudo labels is essential to get public lb=0.747->0.780.
Darn :) I should have run pseudo labels.
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2086102": "Thank you to the hosts and the organizers for having such a challenging competition!\n\nMy solution is based on a CNN. I incorporated various elements, but the following three items were particularly important.\n- **normalization using RobustScaler**\n- **multitask learning of target and frequency**\n- **pseudo labels**\n\n# Data\n\nI used PyFstat to generate data with the same range of timestamps, frequency, and amplitudes as the test data. The signal data was generated with a signal depth (sqrtSX/h0) in the range of [1, 50]. The number of data for each is as follows.\n\n| data type | number of data |\n| ------------------- | -------------- |\n| gap noise | 3000 |\n| nonstationary noise | 2983 |\n| signal | 5400 |\n\n\n\n# Preprocess\n\nI created an image with shape (C,H,W) = (2, 360, 127) by performing the following processing on each of H1 and L1.\n\n- Normalize the square of the amplitudes of H1 and L1 using `sklearn.preprocessing.RobustScaler`.\n - Since the real noise in the test data has large outliers, I use `sklearn.preprocessing.RobustScaler`, which is resistant to outliers.\n- Align H1 and L1 timestamps in the same way as [G2NET large kernel inference](https://www.kaggle.com/code/laeyoung/g2net-large-kernel-inference), resulting in an image with shape (C,H,W) = (2, 360, 5760).\n- Take a moving average to make the image size (2, 360, 5760) -> (2, 360, 127). \n- Clip the values of the channel with the larger maximum value.\n - Because the test set contains data with too large a difference in maximum values between H1 and L1.\n- Finally, I standardize the image with entire data mean and standard deviation.\n\n\n# Model\n\nMultiOutput model predicting target and frequency. \n\n- Target loss is `nn.BCEWithLogitsLoss`\n- Frequency//50 loss is `nn.CrossEntropyLoss` of 11 classes\n- Architecture is tf_efficientnet_b5_ap\n- Change the stride of the first conv layer of the model to (1,2) to scale up the image resolution\n\n```\nclass CustomModel(nn.Module):\n def __init__(self, pretrained=True):\n super().__init__()\n self.net = timm.create_model(\"tf_efficientnet_b5_ap\", \n pretrained=pretrained, \n num_classes=0, \n in_chans=2)\n \n # decrease first conv's stride\n modules_iter = iter(self.net.modules())\n for module in modules_iter:\n if isinstance(module, torch.nn.Conv2d) and tuple(module.stride) == (2, 2):\n break\n module.stride = (1, 2)\n \n # Target \n self.head1 = nn.Sequential(\n nn.Linear(self.net.num_features, 1)\n )\n \n # Frequency//50: 40-500Hz // 50\n freq_div_n = 50\n self.head2 = nn.Sequential(\n nn.Linear(self.net.num_features, (500 // freq_div_n) - (40 // freq_div_n) + 1)\n )\n\n def forward(self, x, labels=None):\n feat = self.net(x)\n y1 = self.head1(feat)\n y2 = self.head2(feat)\n return y1[:,0], y2\n```\n\n# Training\n- cv: target StratifiedKFold(n_splits=5)\n- optimizer: AdamW\n- scheduler: warmup 0-3epoch(lr=4e-6->4e-4) + Cosine Annealing 3-100epoch(lr=4e-4->4e-6)\n\n\n# Augmentation\n\n- Mixup\n - mixup where data with target=1 is not mixed together\n - mixup with the overall mean value of the test set\n- Horizontal and vertical lines of outliers that mimic the real data of the test set\n- [Augmentation of 8th place in the last competition](https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/discussion/275335)（LIGO Swap, Swap with Other Negative）\n- GaussNoise\n- Vertical Shift\n- Torchaudio Masking（time masking, frequency masking）\n- Horizontal Flip, Vertical Flip\n\n\n# Pseudo Labels\n\nPseudo labels using only the simulation data from the test set and pseudo labels excluding real data with amplitudes that take on outliers were effective. \nIt was better to train with soft labels than with hard labels.\n\n\n# TTA\n\nHorizontal Flip, Vertical Flip\n\n \n# Ensemble\n\n[Rank averaging](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205564) of submit.csv files from different models.\nHowever, according to @poteman, Stacking seems to perform slightly better in private lb. (I heard about this after competition. poteman's submit is not used for final submission.)\n\n# P.S.\n**I did all the above solutions by myself. I did not get help from my teammates.**\nCode: https://github.com/riron1206/kaggle-G2Net-Detecting-Continuous-Gravitational-Waves-10th-Place-Solution", "2086297": "Cool, Congrats. I tried RobustScaler then QuantileTransformer to do the same, but both did poorly. I must have done something wrong. Did you try without frequency segmentation? How much boost was pseudo? I planned to do it, but I guessed with so much generated data based upon test data, it probably wouldn't help." } }