I am not a seasoned kaggler or programmer so I might be missing something obvious . So please suggest .
\n\nThe data for the three-four image comp running are huge and there are no way to download necessary data alone via API . How do you get this data in cloud storage ? Downloading locally and uploading to cloud is not feasible due to network problems .
The Kaggle output directory space seems to be a problem for me . Right now most of the SOTA models are big (especially for NLP area) , i cant even run a 5 fold at one shot and run my model . It might crash after running for 5 hours due to output size > 5 GB. The same thing happens when I try to generate dataset from large files e.g. jpg from tiff in this comp . Is there a possibility to increase this local space to lets say 20GB or something ? What is the flipside of that ? How are you all doing it ?
I bought some Google Drive space of 2 TB during DFDC time , but then when i download data via API i understood that it uses the local diskspace while downloading and hence we cant download >100GB at one time . So that space is kind of useless.Is there any workaround ?
I am not a seasoned kaggler or programmer so I might be missing something obvious . So please suggest .
\n\nThe data for the three-four image comp running are huge and there are no way to download necessary data alone via API . How do you get this data in cloud storage ? Downloading locally and uploading to cloud is not feasible due to network problems .
The Kaggle output directory space seems to be a problem for me . Right now most of the SOTA models are big (especially for NLP area) , i cant even run a 5 fold at one shot and run my model . It might crash after running for 5 hours due to output size > 5 GB. The same thing happens when I try to generate dataset from large files e.g. jpg from tiff in this comp . Is there a possibility to increase this local space to lets say 20GB or something ? What is the flipside of that ? How are you all doing it ?
I bought some Google Drive space of 2 TB during DFDC time , but then when i download data via API i understood that it uses the local diskspace while downloading and hence we cant download >100GB at one time . So that space is kind of useless.Is there any workaround ?
I don't understand. Just use the kaggle api to download it in a cloud environment. Make sure you have a large enough ssd or cold disk on your instance. Speaking of prostate cancer competition I have done it once on aws and once on gcp.
Kaggle kernels are very limited yes. Not much you can do about that.
Why are you using Google DRIVE ? Use the real tool for that kind of thing: google cloud.
Alternatively use colab it's probably easier to start with it than starting (costly) instances if you have no experience with it.
", "rawMarkdown": "1. I don't understand. Just use the kaggle api to download it in a cloud environment. Make sure you have a large enough ssd or cold disk on your instance. Speaking of prostate cancer competition I have done it once on aws and once on gcp.\n\n2. Kaggle kernels are very limited yes. Not much you can do about that.\n\n3. Why are you using Google DRIVE ? Use the real tool for that kind of thing: google cloud.\n\nAlternatively use colab it's probably easier to start with it than starting (costly) instances if you have no experience with it.", "votes": -1 } ], "comments": [ { "id": 876661, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-06-06T21:56:41.727000", "content": "I don't understand. Just use the kaggle api to download it in a cloud environment. Make sure you have a large enough ssd or cold disk on your instance. Speaking of prostate cancer competition I have done it once on aws and once on gcp.
Kaggle kernels are very limited yes. Not much you can do about that.
Why are you using Google DRIVE ? Use the real tool for that kind of thing: google cloud.
Alternatively use colab it's probably easier to start with it than starting (costly) instances if you have no experience with it.
", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "875934": "I am not a seasoned kaggler or programmer so I might be missing something obvious . So please suggest . \n\n1. The data for the three-four image comp running are huge and there are no way to download necessary data alone via API . How do you get this data in cloud storage ? Downloading locally and uploading to cloud is not feasible due to network problems .\n\n2. The Kaggle output directory space seems to be a problem for me . Right now most of the SOTA models are big (especially for NLP area) , i cant even run a 5 fold at one shot and run my model . It might crash after running for 5 hours due to output size > 5 GB. The same thing happens when I try to generate dataset from large files e.g. jpg from tiff in this comp . Is there a possibility to increase this local space to lets say 20GB or something ? What is the flipside of that ? How are you all doing it ?\n\n3. I bought some Google Drive space of 2 TB during DFDC time , but then when i download data via API i understood that it uses the local diskspace while downloading and hence we cant download >100GB at one time . So that space is kind of useless.Is there any workaround ?", "876661": "1. I don't understand. Just use the kaggle api to download it in a cloud environment. Make sure you have a large enough ssd or cold disk on your instance. Speaking of prostate cancer competition I have done it once on aws and once on gcp.\n\n2. Kaggle kernels are very limited yes. Not much you can do about that.\n\n3. Why are you using Google DRIVE ? Use the real tool for that kind of thing: google cloud.\n\nAlternatively use colab it's probably easier to start with it than starting (costly) instances if you have no experience with it." } }