![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6227496%2Fbabc06848ffa005059fe3fa33d50ec2c%2Fgoogle_drive.png?generation=1690261545052175&alt=media\")
Hello everyone,
\nI've just joined the competition. The first challenge is how to download such huge data (more than 150 GB)! What I did is that I subscribed for 2TB in Google Drive and installed Google Drive on my local laptop. Unfortunately, the mounted drive didn't show me all the 2TB capacity. It showed me only 200 GB, not 2TB!
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6227496%2Fc01c21ac2b52e8ee71c0062434711446%2Fgoogle_drive_2TB.png?generation=1690261566908281&alt=media\")
My two questions here are,
\nBest Regards,
\nMustafa
Hi @mustafaalshuala
\nFor example, you can run the following code in GoogleColaboratory to download Kaggle data to your GoogleDrive.
import os\nimport time\n\n\n google.colab import drive\ndrive.mount()\n\n\n! pip install --upgrade --force-reinstall --no-deps kaggle > /dev/\n! mkdir ~/.kaggle\n! cp ~/.kaggle/kaggle.json\n! chmod 600 ~/.kaggle/kaggle.json\n\n\nKAGGLE_API_COMMANDS = []\nDOWNLOAD_GOOGLE_DRIVE_PATH = # -> Ex:\n\n\n ==[]:\n API_COMMAND KAGGLE_API_COMMANDS:\n DATASET_PATH = API_COMMAND.split()[-1]\n ZIP_FILE_NAME = DATASET_PATH.split()[-1]\n (f)\n (f)\n\n os.path.exists(DOWNLOAD_GOOGLE_DRIVE_PATH):\n !mkdir {DOWNLOAD_GOOGLE_DRIVE_PATH}\n\n (API_COMMAND.split()):\n !kaggle datasets download -d {DATASET_PATH} -p {DOWNLOAD_GOOGLE_DRIVE_PATH}\n elif (API_COMMAND.split()):\n !kaggle competitions download -c {DATASET_PATH} -p {DOWNLOAD_GOOGLE_DRIVE_PATH}\n\n\n\nSLEEP_TIME = 60\n(f, =); time.sleep(SLEEP_TIME); ();time.sleep(1);\n\n\n ==[]:\n API_COMMAND KAGGLE_API_COMMANDS:\n DATASET_PATH = API_COMMAND.split()[-1]\n ZIP_FILE_NAME = DATASET_PATH.split()[-1]\n (f)\n (f)\n\n !unzip -q {DOWNLOAD_GOOGLE_DRIVE_PATH}/{ZIP_FILE_NAME}.zip -d {DOWNLOAD_GOOGLE_DRIVE_PATH}\n\n\n\nSLEEP_TIME = 60\n(f, =); time.sleep(SLEEP_TIME); ();time.sleep(1);\n\n\nDELETE_ZIP = \n DELETE_ZIP:\n ==[]:\n API_COMMAND KAGGLE_API_COMMANDS:\n DATASET_PATH = API_COMMAND.split()[-1]\n ZIP_FILE_NAME = DATASET_PATH.split()[-1]\n (f)\n (f)\n\n !rm {DOWNLOAD_GOOGLE_DRIVE_PATH}/{ZIP_FILE_NAME}.zip\n\n\n\nimport time\nSLEEP_TIME = 60\n(f, =); time.sleep(SLEEP_TIME); ();time.sleep(1);\n google.colab import runtime\nruntime.unassign()\nThank you K.Mitsuhiro. That helps a lot.
", "rawMarkdown": "Thank you K.Mitsuhiro. That helps a lot." } ] }, { "id": 2357784, "postDate": "2023-07-25T05:52:32.337Z", "content": "Hi @mustafaalshuala , umm maybe you can use kaggle notebooks and from there you can easily access the competition data ( right corner -> add date -> search data name -> press + button) . So you don't have to download all the data. :)
", "rawMarkdown": "Hi @mustafaalshuala , umm maybe you can use kaggle notebooks and from there you can easily access the competition data ( right corner -> add date -> search data name -> press + button) . So you don't have to download all the data. :) ", "votes": 1, "replies": [ { "id": 2379342, "postDate": "2023-08-08T05:57:26.687Z", "content": "Very good suggestion Ayush.
\nmmm.. I bought Colab Pro+ for GPU. Can I use Kaggle notebooks. At the same time, I run on Google resource?
Hello everyone,
\nI've just joined the competition. The first challenge is how to download such huge data (more than 150 GB)! What I did is that I subscribed for 2TB in Google Drive and installed Google Drive on my local laptop. Unfortunately, the mounted drive didn't show me all the 2TB capacity. It showed me only 200 GB, not 2TB!
My two questions here are,
\nBest Regards,
\nMustafa
Hi @mustafaalshuala
\nFor example, you can run the following code in GoogleColaboratory to download Kaggle data to your GoogleDrive.
import os\nimport time\n\n\n google.colab import drive\ndrive.mount()\n\n\n! pip install --upgrade --force-reinstall --no-deps kaggle > /dev/\n! mkdir ~/.kaggle\n! cp ~/.kaggle/kaggle.json\n! chmod 600 ~/.kaggle/kaggle.json\n\n\nKAGGLE_API_COMMANDS = []\nDOWNLOAD_GOOGLE_DRIVE_PATH = # -> Ex:\n\n\n ==[]:\n API_COMMAND KAGGLE_API_COMMANDS:\n DATASET_PATH = API_COMMAND.split()[-1]\n ZIP_FILE_NAME = DATASET_PATH.split()[-1]\n (f)\n (f)\n\n os.path.exists(DOWNLOAD_GOOGLE_DRIVE_PATH):\n !mkdir {DOWNLOAD_GOOGLE_DRIVE_PATH}\n\n (API_COMMAND.split()):\n !kaggle datasets download -d {DATASET_PATH} -p {DOWNLOAD_GOOGLE_DRIVE_PATH}\n elif (API_COMMAND.split()):\n !kaggle competitions download -c {DATASET_PATH} -p {DOWNLOAD_GOOGLE_DRIVE_PATH}\n\n\n\nSLEEP_TIME = 60\n(f, =); time.sleep(SLEEP_TIME); ();time.sleep(1);\n\n\n ==[]:\n API_COMMAND KAGGLE_API_COMMANDS:\n DATASET_PATH = API_COMMAND.split()[-1]\n ZIP_FILE_NAME = DATASET_PATH.split()[-1]\n (f)\n (f)\n\n !unzip -q {DOWNLOAD_GOOGLE_DRIVE_PATH}/{ZIP_FILE_NAME}.zip -d {DOWNLOAD_GOOGLE_DRIVE_PATH}\n\n\n\nSLEEP_TIME = 60\n(f, =); time.sleep(SLEEP_TIME); ();time.sleep(1);\n\n\nDELETE_ZIP = \n DELETE_ZIP:\n ==[]:\n API_COMMAND KAGGLE_API_COMMANDS:\n DATASET_PATH = API_COMMAND.split()[-1]\n ZIP_FILE_NAME = DATASET_PATH.split()[-1]\n (f)\n (f)\n\n !rm {DOWNLOAD_GOOGLE_DRIVE_PATH}/{ZIP_FILE_NAME}.zip\n\n\n\nimport time\nSLEEP_TIME = 60\n(f, =); time.sleep(SLEEP_TIME); ();time.sleep(1);\n google.colab import runtime\nruntime.unassign()\nThank you K.Mitsuhiro. That helps a lot.
", "votes": 0, "replies": [] } ] }, { "id": 2357784, "author_name": "Ayush Kumar", "author_url": "", "post_date": "2023-07-25T05:52:32.337000", "content": "Hi @mustafaalshuala , umm maybe you can use kaggle notebooks and from there you can easily access the competition data ( right corner -> add date -> search data name -> press + button) . So you don't have to download all the data. :)
", "votes": 1, "replies": [ { "id": 2379342, "author_name": "MUSTAFA ALSHUALA", "author_url": "", "post_date": "2023-08-08T05:57:26.687000", "content": "Very good suggestion Ayush.
\nmmm.. I bought Colab Pro+ for GPU. Can I use Kaggle notebooks. At the same time, I run on Google resource?