{
  "id": 433339,
  "title": "How to process large datasets?",
  "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/433339",
  "author_name": "NewToAI_i",
  "post_date": "2023-08-21T10:14:59.583000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hello. The data of this competition is 450GB! can you please tell me how to process this amount of data? do you use TPU in kaggle or use another platform?<br>\nI use GPU100 in kaggle and when i want to train a model on a 3GB image dataset it takes hours (about 10 hours) and it is really difficult and time consuming to do preprocess on images and save them in a dataset for use in other sessions. Can you please help me?</p>",
  "messages": [
    {
      "id": 2400933,
      "postDate": "2023-08-21T10:14:59.583Z",
      "content": "<p>Hello. The data of this competition is 450GB! can you please tell me how to process this amount of data? do you use TPU in kaggle or use another platform?<br>\nI use GPU100 in kaggle and when i want to train a model on a 3GB image dataset it takes hours (about 10 hours) and it is really difficult and time consuming to do preprocess on images and save them in a dataset for use in other sessions. Can you please help me?</p>",
      "rawMarkdown": "Hello. The data of this competition is 450GB! can you please tell me how to process this amount of data? do you use TPU in kaggle or use another platform?\nI use GPU100 in kaggle and when i want to train a model on a 3GB image dataset it takes hours (about 10 hours) and it is really difficult and time consuming to do preprocess on images and save them in a dataset for use in other sessions. Can you please help me?",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2400933": "Hello. The data of this competition is 450GB! can you please tell me how to process this amount of data? do you use TPU in kaggle or use another platform?\nI use GPU100 in kaggle and when i want to train a model on a 3GB image dataset it takes hours (about 10 hours) and it is really difficult and time consuming to do preprocess on images and save them in a dataset for use in other sessions. Can you please help me?"
  }
}