Hi everyone,
\nI'm having trouble getting the data downloaded, as the zip file size is too large. I'm using Colab at the moment, so is there a way for me to download the data from Kaggle straight to Google Drive without eating up Colab's disk space? I downloaded the data using Kaggle's API command
\n!kaggle competitions download -c vinbigdata-chest-xray-abnormalities-detection -p \"/content/drive/My Drive/kaggle/xrayproject\", but there is not enough space on Colab's disk for me to finish downloading it to Drive.
The Competition Dataset size is around 190 GB. Google colab disk will run out of memory on loading the dataset.
\nThankfully, most of space is taken up by huge dicom images. We can convert these to png format and save as datasets.
\nThis discussion has links to datasets in Png format in various sizes. They will fit into Google Colab disk easily.
And since you plan on using Colab, having these uploaded once to your drive should be helpful for the comps duration
", "rawMarkdown": "And since you plan on using Colab, having these uploaded once to your drive should be helpful for the comps duration", "votes": 1 } ] }, { "id": 1133964, "postDate": "2020-12-31T17:37:36.183Z", "content": "Hi everyone,
\nI'm having trouble getting the data downloaded, as the zip file size is too large. I'm using Colab at the moment, so is there a way for me to download the data from Kaggle straight to Google Drive without eating up Colab's disk space? I downloaded the data using Kaggle's API command
\n!kaggle competitions download -c vinbigdata-chest-xray-abnormalities-detection -p \"/content/drive/My Drive/kaggle/xrayproject\", but there is not enough space on Colab's disk for me to finish downloading it to Drive.
Unfortunately my laptop also has fairly limited disk space, and this is my second project this month which is why I moved my project to Colab thinking it will work out (normally I work with Jupyter Notebooks). I might have to download it bit by bit to my PC and then transfer some at a time.
", "rawMarkdown": "Unfortunately my laptop also has fairly limited disk space, and this is my second project this month which is why I moved my project to Colab thinking it will work out (normally I work with Jupyter Notebooks). I might have to download it bit by bit to my PC and then transfer some at a time." } ], "comments": [ { "id": 1134157, "author_name": "Yerram Varun", "author_url": "", "post_date": "2020-12-31T22:51:09.920000", "content": "The Competition Dataset size is around 190 GB. Google colab disk will run out of memory on loading the dataset.
\nThankfully, most of space is taken up by huge dicom images. We can convert these to png format and save as datasets.
\nThis discussion has links to datasets in Png format in various sizes. They will fit into Google Colab disk easily.
And since you plan on using Colab, having these uploaded once to your drive should be helpful for the comps duration
", "votes": 1, "replies": [] } ] }, { "id": 1134207, "author_name": "Phan Nguyen", "author_url": "", "post_date": "2021-01-01T02:07:49.960000", "content": "Unfortunately my laptop also has fairly limited disk space, and this is my second project this month which is why I moved my project to Colab thinking it will work out (normally I work with Jupyter Notebooks). I might have to download it bit by bit to my PC and then transfer some at a time.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1134157": "The Competition Dataset size is around 190 GB. Google colab disk will run out of memory on loading the dataset.\nThankfully, most of space is taken up by huge dicom images. We can convert these to png format and save as datasets.\n[This](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/207859) discussion has links to datasets in Png format in various sizes. They will fit into Google Colab disk easily.", "1133964": "Hi everyone,\n\nI'm having trouble getting the data downloaded, as the zip file size is too large. I'm using Colab at the moment, so is there a way for me to download the data from Kaggle straight to Google Drive without eating up Colab's disk space? I downloaded the data using Kaggle's API command \n*!kaggle competitions download -c vinbigdata-chest-xray-abnormalities-detection -p \"/content/drive/My Drive/kaggle/xrayproject\"*, but there is not enough space on Colab's disk for me to finish downloading it to Drive. ", "1134207": "Unfortunately my laptop also has fairly limited disk space, and this is my second project this month which is why I moved my project to Colab thinking it will work out (normally I work with Jupyter Notebooks). I might have to download it bit by bit to my PC and then transfer some at a time." } }