{ "id": 191457, "title": "How to break 9 hours notebook limit?", "url": "/competitions/rsna-str-pulmonary-embolism-detection/discussion/191457", "author_name": "Dewei Chen", "post_date": "2020-10-16T15:29:11.188000", "votes": 6, "comment_count": 6, "views": 0, "content": "

I want to training a model by using notebook with GPU, but it seems follows this rule:

\n
\n

CPU Notebook <= 9 hours run-time
\n GPU Notebook <= 9 hours run-time

\n
\n

So my notebook automatics stop after 9 hours later. But my model needs about 20hours.
\nAnyone knows how to break this 9 hours limit in notebook?

", "messages": [ { "id": 1051515, "postDate": "2020-10-16T15:29:11.190Z", "content": "

I want to training a model by using notebook with GPU, but it seems follows this rule:

\n
\n

CPU Notebook <= 9 hours run-time
\n GPU Notebook <= 9 hours run-time

\n
\n

So my notebook automatics stop after 9 hours later. But my model needs about 20hours.
\nAnyone knows how to break this 9 hours limit in notebook?

", "rawMarkdown": "I want to training a model by using notebook with GPU, but it seems follows this rule:\n\n> CPU Notebook <= 9 hours run-time\n> GPU Notebook <= 9 hours run-time\n\nSo my notebook automatics stop after 9 hours later. But my model needs about 20hours.\nAnyone knows how to break this 9 hours limit in notebook?", "votes": 6 }, { "id": 1051539, "postDate": "2020-10-16T16:00:58.850Z", "content": "

I found Kaggle document here:

\n
\n

Technical Specifications
\n Kaggle Notebooks run in a remote computational environment. We provide the hardware—you need only worry about the code.

\n

At time of writing, each Notebook editing session is provided with the following resources:

\n

9 hours execution time (note: TPU sessions currently have a 3 hour limit)

\n

5 Gigabytes of auto-saved disk space (/kaggle/working)

\n

Additional scratchpad disk space (outside /kaggle/working) that will not be saved outside of the current session

\n
\n

9 hours is limit, but we can using 2 notebooks save and load to keep training. Thanks the common.

", "rawMarkdown": "I found Kaggle document here:\n\n> Technical Specifications\n> Kaggle Notebooks run in a remote computational environment. We provide the hardware—you need only worry about the code.\n\n> At time of writing, each Notebook editing session is provided with the following resources:\n\n> 9 hours execution time (note: TPU sessions currently have a 3 hour limit)\n\n> 5 Gigabytes of auto-saved disk space (/kaggle/working)\n\n> Additional scratchpad disk space (outside /kaggle/working) that will not be saved outside of the current session\n\n9 hours is limit, but we can using **2 notebooks save and load** to keep training. Thanks the common.", "votes": 1 }, { "id": 1051530, "postDate": "2020-10-16T15:51:41.153Z", "content": "

They're there for a reason so I'm pretty sure you can't bypass that limit. You might be able it split it into multiple notebooks for script runs, but that won't work during submission to a competition.

", "rawMarkdown": "They're there for a reason so I'm pretty sure you can't bypass that limit. You might be able it split it into multiple notebooks for script runs, but that won't work during submission to a competition.", "votes": 1, "replies": [ { "id": 1051532, "postDate": "2020-10-16T15:54:40.783Z", "content": "

Yes, as @underwearfitting said. Agree with your idea.

", "rawMarkdown": "Yes, as @underwearfitting said. Agree with your idea." } ] }, { "id": 1051523, "postDate": "2020-10-16T15:42:34.213Z", "content": "

Ok, then use 2 notebooks. Train Save then load.

", "rawMarkdown": "Ok, then use 2 notebooks. Train Save then load.", "votes": 2, "replies": [ { "id": 1051527, "postDate": "2020-10-16T15:50:29.917Z", "content": "

maybe a good idea, have you done this?

", "rawMarkdown": "maybe a good idea, have you done this?" }, { "id": 1051815, "postDate": "2020-10-17T00:03:57.410Z", "content": "

Sin is right - I have done this with colab - do a timer at the start, and if it reaches 8.5 hours, save a model. Then, in a different notebook, load the saved model and continue training.

", "rawMarkdown": "Sin is right - I have done this with colab - do a timer at the start, and if it reaches 8.5 hours, save a model. Then, in a different notebook, load the saved model and continue training.", "votes": 2 } ] } ], "comments": [ { "id": 1051539, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-16T16:00:58.850000", "content": "

I found Kaggle document here:

\n
\n

Technical Specifications
\n Kaggle Notebooks run in a remote computational environment. We provide the hardware—you need only worry about the code.

\n

At time of writing, each Notebook editing session is provided with the following resources:

\n

9 hours execution time (note: TPU sessions currently have a 3 hour limit)

\n

5 Gigabytes of auto-saved disk space (/kaggle/working)

\n

Additional scratchpad disk space (outside /kaggle/working) that will not be saved outside of the current session

\n
\n

9 hours is limit, but we can using 2 notebooks save and load to keep training. Thanks the common.

", "votes": 1, "replies": [] }, { "id": 1051530, "author_name": "Rupesh Deshmukh", "author_url": "", "post_date": "2020-10-16T15:51:41.153000", "content": "

They're there for a reason so I'm pretty sure you can't bypass that limit. You might be able it split it into multiple notebooks for script runs, but that won't work during submission to a competition.

", "votes": 1, "replies": [ { "id": 1051532, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-16T15:54:40.783000", "content": "

Yes, as @underwearfitting said. Agree with your idea.

", "votes": 0, "replies": [] } ] }, { "id": 1051523, "author_name": "sin", "author_url": "", "post_date": "2020-10-16T15:42:34.213000", "content": "

Ok, then use 2 notebooks. Train Save then load.

", "votes": 2, "replies": [ { "id": 1051527, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-10-16T15:50:29.917000", "content": "

maybe a good idea, have you done this?

", "votes": 0, "replies": [] }, { "id": 1051815, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-17T00:03:57.410000", "content": "

Sin is right - I have done this with colab - do a timer at the start, and if it reaches 8.5 hours, save a model. Then, in a different notebook, load the saved model and continue training.

", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "1051515": "I want to training a model by using notebook with GPU, but it seems follows this rule:\n\n> CPU Notebook <= 9 hours run-time\n> GPU Notebook <= 9 hours run-time\n\nSo my notebook automatics stop after 9 hours later. But my model needs about 20hours.\nAnyone knows how to break this 9 hours limit in notebook?", "1051539": "I found Kaggle document here:\n\n> Technical Specifications\n> Kaggle Notebooks run in a remote computational environment. We provide the hardware—you need only worry about the code.\n\n> At time of writing, each Notebook editing session is provided with the following resources:\n\n> 9 hours execution time (note: TPU sessions currently have a 3 hour limit)\n\n> 5 Gigabytes of auto-saved disk space (/kaggle/working)\n\n> Additional scratchpad disk space (outside /kaggle/working) that will not be saved outside of the current session\n\n9 hours is limit, but we can using **2 notebooks save and load** to keep training. Thanks the common.", "1051530": "They're there for a reason so I'm pretty sure you can't bypass that limit. You might be able it split it into multiple notebooks for script runs, but that won't work during submission to a competition.", "1051523": "Ok, then use 2 notebooks. Train Save then load." } }