{
  "id": 430894,
  "title": "Has anybody used Self-Supervised Learning in this competition? Success or fail?",
  "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/430894",
  "author_name": "Remek Kinas",
  "post_date": "2023-08-11T08:12:28.633000",
  "votes": 8,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi, <br>\nI'm curious if any of you used SSL for the initial pre-training of the encoder model?  From what I can see TOP10 solutions, no one mentions the use of the remaining data (those except 4 in the sequence) for pre-training using SSL.</p>\n<p>Personally, during the competition, I conducted tests with three methods:</p>\n<ul>\n<li>Bootstrap Your Own Latent</li>\n<li>Momentum Contrast</li>\n<li>Simple Siamese</li>\n</ul>\n<p>Unfortunately, the results I got were worse (on the CV) than those from the imagenet checkpoint. What are your observations? Has anyone used SSL and it produced the expected results? Hypotheses why it didn't work on this DS?</p>\n<p>Of course, I will still check this method after analyzing all TOP10 solutions.</p>",
  "messages": [
    {
      "id": 2385233,
      "postDate": "2023-08-11T08:12:28.633Z",
      "content": "<p>Hi, <br>\nI'm curious if any of you used SSL for the initial pre-training of the encoder model?  From what I can see TOP10 solutions, no one mentions the use of the remaining data (those except 4 in the sequence) for pre-training using SSL.</p>\n<p>Personally, during the competition, I conducted tests with three methods:</p>\n<ul>\n<li>Bootstrap Your Own Latent</li>\n<li>Momentum Contrast</li>\n<li>Simple Siamese</li>\n</ul>\n<p>Unfortunately, the results I got were worse (on the CV) than those from the imagenet checkpoint. What are your observations? Has anyone used SSL and it produced the expected results? Hypotheses why it didn't work on this DS?</p>\n<p>Of course, I will still check this method after analyzing all TOP10 solutions.</p>",
      "rawMarkdown": "Hi, \nI'm curious if any of you used SSL for the initial pre-training of the encoder model?  From what I can see TOP10 solutions, no one mentions the use of the remaining data (those except 4 in the sequence) for pre-training using SSL.\n\nPersonally, during the competition, I conducted tests with three methods:\n- Bootstrap Your Own Latent\n- Momentum Contrast\n- Simple Siamese\n\nUnfortunately, the results I got were worse (on the CV) than those from the imagenet checkpoint. What are your observations? Has anyone used SSL and it produced the expected results? Hypotheses why it didn't work on this DS?\n\nOf course, I will still check this method after analyzing all TOP10 solutions.",
      "votes": 7
    },
    {
      "id": 2386857,
      "postDate": "2023-08-12T07:35:45.407Z",
      "content": "<p>i am doing it for post submission. you will have to wait for a week or two to see my post on the results.<br>\nmy target are:</p>\n<ol>\n<li>covnextv2 convolutional MAE</li>\n</ol>\n<p>2.video MAE</p>\n<p>there are many frames without label. only frame 4 has label.</p>\n<p>i want to compare the following:</p>\n<ol>\n<li>single frame model trained on frame 4 </li>\n<li>single frame model trained on frame 4  + SSL on other frames</li>\n<li>single frame model trained on frame 4  + pedso-label on other frames</li>\n</ol>\n<p>for mult-frame it would be similar, except we now have an additional option of video MAE </p>",
      "rawMarkdown": "i am doing it for post submission. you will have to wait for a week or two to see my post on the results.\nmy target are:\n\n1. covnextv2 convolutional MAE\n\n2.video MAE\n\nthere are many frames without label. only frame 4 has label.\n\ni want to compare the following:\n1. single frame model trained on frame 4 \n2. single frame model trained on frame 4  + SSL on other frames\n2. single frame model trained on frame 4  + pedso-label on other frames\n\nfor mult-frame it would be similar, except we now have an additional option of video MAE \n",
      "votes": 4
    },
    {
      "id": 2386953,
      "postDate": "2023-08-12T09:23:30.860Z",
      "content": "<p>I've shared the PL masks and images I scrapped from the GOES16 bucket if you wanna try pretraining on that. I don't think I could reproduce exactly what the hosts did but it looks similar. </p>\n<p>Iafoss used them for pretraining (in a 2D setup), and I randomly sampled PL images during training of my models using RandomCropWithMasks.<br>\nBoth strategies were alright although it hurt models diversity. We did use one NextVit with pretraining in the final ensemble and it was better on private than the non-pretrained one.</p>\n<p>Links : </p>\n<ul>\n<li><a href=\"https://www.kaggle.com/datasets/theoviel/contrails-goes16-mask-1\" target=\"_blank\">https://www.kaggle.com/datasets/theoviel/contrails-goes16-mask-1</a></li>\n<li><a href=\"https://www.kaggle.com/datasets/theoviel/contrails-goes16-img-1\" target=\"_blank\">https://www.kaggle.com/datasets/theoviel/contrails-goes16-img-1</a></li>\n</ul>",
      "rawMarkdown": "I've shared the PL masks and images I scrapped from the GOES16 bucket if you wanna try pretraining on that. I don't think I could reproduce exactly what the hosts did but it looks similar. \n\nIafoss used them for pretraining (in a 2D setup), and I randomly sampled PL images during training of my models using RandomCropWithMasks.\nBoth strategies were alright although it hurt models diversity. We did use one NextVit with pretraining in the final ensemble and it was better on private than the non-pretrained one.\n\nLinks : \n- https://www.kaggle.com/datasets/theoviel/contrails-goes16-mask-1\n- https://www.kaggle.com/datasets/theoviel/contrails-goes16-img-1",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 2386857,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2023-08-12T07:35:45.407000",
      "content": "<p>i am doing it for post submission. you will have to wait for a week or two to see my post on the results.<br>\nmy target are:</p>\n<ol>\n<li>covnextv2 convolutional MAE</li>\n</ol>\n<p>2.video MAE</p>\n<p>there are many frames without label. only frame 4 has label.</p>\n<p>i want to compare the following:</p>\n<ol>\n<li>single frame model trained on frame 4 </li>\n<li>single frame model trained on frame 4  + SSL on other frames</li>\n<li>single frame model trained on frame 4  + pedso-label on other frames</li>\n</ol>\n<p>for mult-frame it would be similar, except we now have an additional option of video MAE </p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 2386953,
      "author_name": "Theo Viel",
      "author_url": "",
      "post_date": "2023-08-12T09:23:30.860000",
      "content": "<p>I've shared the PL masks and images I scrapped from the GOES16 bucket if you wanna try pretraining on that. I don't think I could reproduce exactly what the hosts did but it looks similar. </p>\n<p>Iafoss used them for pretraining (in a 2D setup), and I randomly sampled PL images during training of my models using RandomCropWithMasks.<br>\nBoth strategies were alright although it hurt models diversity. We did use one NextVit with pretraining in the final ensemble and it was better on private than the non-pretrained one.</p>\n<p>Links : </p>\n<ul>\n<li><a href=\"https://www.kaggle.com/datasets/theoviel/contrails-goes16-mask-1\" target=\"_blank\">https://www.kaggle.com/datasets/theoviel/contrails-goes16-mask-1</a></li>\n<li><a href=\"https://www.kaggle.com/datasets/theoviel/contrails-goes16-img-1\" target=\"_blank\">https://www.kaggle.com/datasets/theoviel/contrails-goes16-img-1</a></li>\n</ul>",
      "votes": 2,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2385233": "Hi, \nI'm curious if any of you used SSL for the initial pre-training of the encoder model?  From what I can see TOP10 solutions, no one mentions the use of the remaining data (those except 4 in the sequence) for pre-training using SSL.\n\nPersonally, during the competition, I conducted tests with three methods:\n- Bootstrap Your Own Latent\n- Momentum Contrast\n- Simple Siamese\n\nUnfortunately, the results I got were worse (on the CV) than those from the imagenet checkpoint. What are your observations? Has anyone used SSL and it produced the expected results? Hypotheses why it didn't work on this DS?\n\nOf course, I will still check this method after analyzing all TOP10 solutions.",
    "2386857": "i am doing it for post submission. you will have to wait for a week or two to see my post on the results.\nmy target are:\n\n1. covnextv2 convolutional MAE\n\n2.video MAE\n\nthere are many frames without label. only frame 4 has label.\n\ni want to compare the following:\n1. single frame model trained on frame 4 \n2. single frame model trained on frame 4  + SSL on other frames\n2. single frame model trained on frame 4  + pedso-label on other frames\n\nfor mult-frame it would be similar, except we now have an additional option of video MAE \n",
    "2386953": "I've shared the PL masks and images I scrapped from the GOES16 bucket if you wanna try pretraining on that. I don't think I could reproduce exactly what the hosts did but it looks similar. \n\nIafoss used them for pretraining (in a 2D setup), and I randomly sampled PL images during training of my models using RandomCropWithMasks.\nBoth strategies were alright although it hurt models diversity. We did use one NextVit with pretraining in the final ensemble and it was better on private than the non-pretrained one.\n\nLinks : \n- https://www.kaggle.com/datasets/theoviel/contrails-goes16-mask-1\n- https://www.kaggle.com/datasets/theoviel/contrails-goes16-img-1"
  }
}