{
  "id": 416840,
  "title": "Do you download large datasets?",
  "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/416840",
  "author_name": "Kelvin Lim",
  "post_date": "2023-06-13T06:40:45.922000",
  "votes": 1,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hi there,</p>\n<p>I'm fairly new to Kaggle and it's my first time doing a competition with such a heavy dataset.</p>\n<p>I'm wondering if all participants are downloading the datasets on their local and then loading them on notebooks in batches (for RAM purposes)? Or are people leveraging cloud storage/google drive/colab?</p>\n<p>The reason I'm asking is because my current laptop only has 256GB and the dataset itself is &gt;300GB. Even the max limit on google drive is 100GB.</p>\n<p>NB: Is it fundamental to have a high-performing and high storage computer to participate in those competitions? Or is there any alternative using the cloud?</p>\n<p>Cheers :)</p>",
  "messages": [
    {
      "id": 2300721,
      "postDate": "2023-06-13T11:43:16.113Z",
      "content": "<p>Hello <a href=\"https://www.kaggle.com/kelvinlim\" target=\"_blank\">@kelvinlim</a>,<br>\nThis might not be the perfect answer for your question, but it will most likely help.<br>\nFirst off, don't train models on your laptop GPU, it will either take weeks or burn your GPU (I assume your laptop isn't the most powerful since it has 256GB or storage). <br>\nSecondly, there is a way to reduce this dataset to 18GB and achieve correct LB (Thats what i'm using rn and I have a correct LB score):<br>\nOpen a kaggle kernel and create there a dataset that only consist of the 5th sequence(the ones the labelers saw) and only with the 3bands necessary to make the ASHRGB image. This will take 16GB to which you add the labels. <br>\nAnd there you have it, a 18GB dataset with which you can train competitive models.</p>",
      "rawMarkdown": "Hello @kelvinlim,\nThis might not be the perfect answer for your question, but it will most likely help.\nFirst off, don't train models on your laptop GPU, it will either take weeks or burn your GPU (I assume your laptop isn't the most powerful since it has 256GB or storage). \nSecondly, there is a way to reduce this dataset to 18GB and achieve correct LB (Thats what i'm using rn and I have a correct LB score):\nOpen a kaggle kernel and create there a dataset that only consist of the 5th sequence(the ones the labelers saw) and only with the 3bands necessary to make the ASHRGB image. This will take 16GB to which you add the labels. \nAnd there you have it, a 18GB dataset with which you can train competitive models.",
      "votes": 1,
      "replies": [
        {
          "id": 2300727,
          "postDate": "2023-06-13T11:47:29.397Z",
          "content": "<p>And to answer your NB, no, you don't need a high-performing and high storage computer, even tho it greatly helps. I believe a lot of top kagglers bought there DL machine with competition earnings.</p>",
          "rawMarkdown": "And to answer your NB, no, you don't need a high-performing and high storage computer, even tho it greatly helps. I believe a lot of top kagglers bought there DL machine with competition earnings.",
          "votes": 1,
          "replies": [
            {
              "id": 2307201,
              "postDate": "2023-06-18T01:23:26.190Z",
              "content": "<p>I too am having trouble downloading the dataset. is it possible to download the dataset to a local PC as long as it is smaller to 18GB?</p>",
              "rawMarkdown": "I too am having trouble downloading the dataset. is it possible to download the dataset to a local PC as long as it is smaller to 18GB?"
            },
            {
              "id": 2307636,
              "postDate": "2023-06-18T10:50:46.530Z",
              "content": "<p>You can download it the way you want. But you can acheive good LB results with 18GB only.</p>",
              "rawMarkdown": "You can download it the way you want. But you can acheive good LB results with 18GB only."
            },
            {
              "id": 2308482,
              "postDate": "2023-06-19T02:08:36.650Z",
              "content": "<p>Is that true?</p>\n<p>When I tried to download from my browser, an error occurred and the download did not complete.<br>\n(Do I need to download from the kaggle command?)</p>",
              "rawMarkdown": "Is that true?\n\nWhen I tried to download from my browser, an error occurred and the download did not complete.\n(Do I need to download from the kaggle command?)"
            }
          ]
        }
      ]
    },
    {
      "id": 2301040,
      "postDate": "2023-06-13T15:28:35.833Z",
      "content": "<p>The train.zip is 291 GB. <br>\nYou can convert it through \"Otsu\" (or a simple XGB) to have &lt; 4.5 GB.<br>\nThat should be manageable even for a laptop.</p>",
      "rawMarkdown": "The train.zip is 291 GB. \nYou can convert it through \"Otsu\" (or a simple XGB) to have < 4.5 GB.\nThat should be manageable even for a laptop."
    },
    {
      "id": 2300344,
      "postDate": "2023-06-13T06:40:45.923Z",
      "content": "<p>Hi there,</p>\n<p>I'm fairly new to Kaggle and it's my first time doing a competition with such a heavy dataset.</p>\n<p>I'm wondering if all participants are downloading the datasets on their local and then loading them on notebooks in batches (for RAM purposes)? Or are people leveraging cloud storage/google drive/colab?</p>\n<p>The reason I'm asking is because my current laptop only has 256GB and the dataset itself is &gt;300GB. Even the max limit on google drive is 100GB.</p>\n<p>NB: Is it fundamental to have a high-performing and high storage computer to participate in those competitions? Or is there any alternative using the cloud?</p>\n<p>Cheers :)</p>",
      "rawMarkdown": "Hi there,\n\nI'm fairly new to Kaggle and it's my first time doing a competition with such a heavy dataset.\n\nI'm wondering if all participants are downloading the datasets on their local and then loading them on notebooks in batches (for RAM purposes)? Or are people leveraging cloud storage/google drive/colab?\n\nThe reason I'm asking is because my current laptop only has 256GB and the dataset itself is >300GB. Even the max limit on google drive is 100GB.\n\nNB: Is it fundamental to have a high-performing and high storage computer to participate in those competitions? Or is there any alternative using the cloud?\n\nCheers :)"
    },
    {
      "id": 2300628,
      "postDate": "2023-06-13T10:07:18.140Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2300721,
      "author_name": "JEANMPIA",
      "author_url": "",
      "post_date": "2023-06-13T11:43:16.113000",
      "content": "<p>Hello <a href=\"https://www.kaggle.com/kelvinlim\" target=\"_blank\">@kelvinlim</a>,<br>\nThis might not be the perfect answer for your question, but it will most likely help.<br>\nFirst off, don't train models on your laptop GPU, it will either take weeks or burn your GPU (I assume your laptop isn't the most powerful since it has 256GB or storage). <br>\nSecondly, there is a way to reduce this dataset to 18GB and achieve correct LB (Thats what i'm using rn and I have a correct LB score):<br>\nOpen a kaggle kernel and create there a dataset that only consist of the 5th sequence(the ones the labelers saw) and only with the 3bands necessary to make the ASHRGB image. This will take 16GB to which you add the labels. <br>\nAnd there you have it, a 18GB dataset with which you can train competitive models.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2300727,
          "author_name": "JEANMPIA",
          "author_url": "",
          "post_date": "2023-06-13T11:47:29.397000",
          "content": "<p>And to answer your NB, no, you don't need a high-performing and high storage computer, even tho it greatly helps. I believe a lot of top kagglers bought there DL machine with competition earnings.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2307201,
              "author_name": "tanako-",
              "author_url": "",
              "post_date": "2023-06-18T01:23:26.190000",
              "content": "<p>I too am having trouble downloading the dataset. is it possible to download the dataset to a local PC as long as it is smaller to 18GB?</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2307636,
              "author_name": "JEANMPIA",
              "author_url": "",
              "post_date": "2023-06-18T10:50:46.530000",
              "content": "<p>You can download it the way you want. But you can acheive good LB results with 18GB only.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2308482,
              "author_name": "tanako-",
              "author_url": "",
              "post_date": "2023-06-19T02:08:36.650000",
              "content": "<p>Is that true?</p>\n<p>When I tried to download from my browser, an error occurred and the download did not complete.<br>\n(Do I need to download from the kaggle command?)</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2301040,
      "author_name": "Tord Malmgren",
      "author_url": "",
      "post_date": "2023-06-13T15:28:35.833000",
      "content": "<p>The train.zip is 291 GB. <br>\nYou can convert it through \"Otsu\" (or a simple XGB) to have &lt; 4.5 GB.<br>\nThat should be manageable even for a laptop.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2300628,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-06-13T10:07:18.140000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2300721": "Hello @kelvinlim,\nThis might not be the perfect answer for your question, but it will most likely help.\nFirst off, don't train models on your laptop GPU, it will either take weeks or burn your GPU (I assume your laptop isn't the most powerful since it has 256GB or storage). \nSecondly, there is a way to reduce this dataset to 18GB and achieve correct LB (Thats what i'm using rn and I have a correct LB score):\nOpen a kaggle kernel and create there a dataset that only consist of the 5th sequence(the ones the labelers saw) and only with the 3bands necessary to make the ASHRGB image. This will take 16GB to which you add the labels. \nAnd there you have it, a 18GB dataset with which you can train competitive models.",
    "2301040": "The train.zip is 291 GB. \nYou can convert it through \"Otsu\" (or a simple XGB) to have < 4.5 GB.\nThat should be manageable even for a laptop.",
    "2300344": "Hi there,\n\nI'm fairly new to Kaggle and it's my first time doing a competition with such a heavy dataset.\n\nI'm wondering if all participants are downloading the datasets on their local and then loading them on notebooks in batches (for RAM purposes)? Or are people leveraging cloud storage/google drive/colab?\n\nThe reason I'm asking is because my current laptop only has 256GB and the dataset itself is >300GB. Even the max limit on google drive is 100GB.\n\nNB: Is it fundamental to have a high-performing and high storage computer to participate in those competitions? Or is there any alternative using the cloud?\n\nCheers :)",
    "2300628": ""
  }
}