Joined this competition late and was wondering what is the average training time for everyone. My model seems to be running for last 23 hours and is at 0.93 CV now.
", "messages": [ { "id": 427446, "postDate": "2018-11-25T14:22:42.940Z", "content": "22 hours for 1 epoch, model converge spend about 4 to 5 days.....
", "rawMarkdown": "22 hours for 1 epoch, model converge spend about 4 to 5 days.....", "votes": 3, "replies": [ { "id": 427916, "postDate": "2018-11-26T11:45:56.400Z", "content": "Which gpu do you use?
", "rawMarkdown": "Which gpu do you use?" } ] }, { "id": 427430, "postDate": "2018-11-25T13:46:24.350Z", "content": "Joined this competition late and was wondering what is the average training time for everyone. My model seems to be running for last 23 hours and is at 0.93 CV now.
", "rawMarkdown": "Joined this competition late and was wondering what is the average training time for everyone. My model seems to be running for last 23 hours and is at 0.93 CV now.", "votes": 1 }, { "id": 428836, "postDate": "2018-11-28T00:37:28.510Z", "content": "70 hours per epoch for fine tuning on 6*K80.......
", "rawMarkdown": "70 hours per epoch for fine tuning on 6*K80......." }, { "id": 428600, "postDate": "2018-11-27T14:51:37.600Z", "content": "I've made my data generator such that I can use a subset of classes (for quicker experimentation with a smaller dataset).
\n\nhttps://github.com/colinator/doodle_generator/blob/master/data_generator_uniform_final.ipynb
\n\nBut, it's revealed to me that I probably can't afford this contest. To run 5 epochs through 100,000 samples of 34 classes, with a mobilenet, takes ~3 hours. So with all 340 classes, it'll take ~30 hours. Paperspace charges $0.78/hour for a nice gpu machine. Wish I had more hardware!
", "rawMarkdown": "I've made my data generator such that I can use a subset of classes (for quicker experimentation with a smaller dataset).\n\nhttps://github.com/colinator/doodle_generator/blob/master/data_generator_uniform_final.ipynb\n\nBut, it's revealed to me that I probably can't afford this contest. To run 5 epochs through 100,000 samples of 34 classes, with a mobilenet, takes ~3 hours. So with all 340 classes, it'll take ~30 hours. Paperspace charges $0.78/hour for a nice gpu machine. Wish I had more hardware!", "replies": [ { "id": 428646, "postDate": "2018-11-27T16:50:35.193Z", "content": "you can use $300 google cloud credits if you haven't used it yet
", "rawMarkdown": "you can use $300 google cloud credits if you haven't used it yet", "votes": 1 }, { "id": 428791, "postDate": "2018-11-27T22:32:12.940Z", "content": "Oof, trying to do this now. GCP is a maze. I get various errors like \"Failed to load: tracking number xxx\", and \"This API cannot be enabled at the moment. You may lack appropriate permissions.\", even though my gcp account is set up. Directions to run jupyter involve running bash scripts, seriously? https://cloud.google.com/dataproc/docs/tutorials/jupyter-notebook? Not sure what's current here and what's obsolete...\nI understand google collab is free, but don't like the 12 hour thing. What do you folks use on GCP? I hate devops.
", "rawMarkdown": "Oof, trying to do this now. GCP is a maze. I get various errors like \"Failed to load: tracking number xxx\", and \"This API cannot be enabled at the moment. You may lack appropriate permissions.\", even though my gcp account is set up. Directions to run jupyter involve running bash scripts, seriously? https://cloud.google.com/dataproc/docs/tutorials/jupyter-notebook? Not sure what's current here and what's obsolete...\nI understand google collab is free, but don't like the 12 hour thing. What do you folks use on GCP? I hate devops." }, { "id": 428911, "postDate": "2018-11-28T04:13:23.217Z", "content": "I use fastai installation method. It worked for me in the past so i never bothered to waste precious hours on changing that.
", "rawMarkdown": "I use fastai installation method. It worked for me in the past so i never bothered to waste precious hours on changing that." } ] }, { "id": 427606, "postDate": "2018-11-25T20:34:53.727Z", "content": "Not doing full data set each epoch, but my final model will have run for almost 150 hours when it stops sometime this evening. This is an older PC but is running with 3 GPU's.
", "rawMarkdown": "Not doing full data set each epoch, but my final model will have run for almost 150 hours when it stops sometime this evening. This is an older PC but is running with 3 GPU's." }, { "id": 427591, "postDate": "2018-11-25T19:43:55.880Z", "content": "2 epochs of 7 hours got me lb 935. I guess I need to go deeper or reduce learning rate...
", "rawMarkdown": "2 epochs of 7 hours got me lb 935. I guess I need to go deeper or reduce learning rate..." }, { "id": 427515, "postDate": "2018-11-25T17:01:56.153Z", "content": "This highly depends on your model and your GPU but I think 1 epoch takes 1-2 days is normal
", "rawMarkdown": "This highly depends on your model and your GPU but I think 1 epoch takes 1-2 days is normal" } ], "comments": [ { "id": 427446, "author_name": "wh1te", "author_url": "", "post_date": "2018-11-25T14:22:42.940000", "content": "22 hours for 1 epoch, model converge spend about 4 to 5 days.....
", "votes": 3, "replies": [ { "id": 427916, "author_name": "hahaha", "author_url": "", "post_date": "2018-11-26T11:45:56.400000", "content": "Which gpu do you use?
", "votes": 0, "replies": [] } ] }, { "id": 428836, "author_name": "Peiyuan Liao", "author_url": "", "post_date": "2018-11-28T00:37:28.510000", "content": "70 hours per epoch for fine tuning on 6*K80.......
", "votes": 0, "replies": [] }, { "id": 428600, "author_name": "Colin", "author_url": "", "post_date": "2018-11-27T14:51:37.600000", "content": "I've made my data generator such that I can use a subset of classes (for quicker experimentation with a smaller dataset).
\n\nhttps://github.com/colinator/doodle_generator/blob/master/data_generator_uniform_final.ipynb
\n\nBut, it's revealed to me that I probably can't afford this contest. To run 5 epochs through 100,000 samples of 34 classes, with a mobilenet, takes ~3 hours. So with all 340 classes, it'll take ~30 hours. Paperspace charges $0.78/hour for a nice gpu machine. Wish I had more hardware!
", "votes": 0, "replies": [ { "id": 428646, "author_name": "Ankit Sati", "author_url": "", "post_date": "2018-11-27T16:50:35.193000", "content": "you can use $300 google cloud credits if you haven't used it yet
", "votes": 1, "replies": [] }, { "id": 428791, "author_name": "Colin", "author_url": "", "post_date": "2018-11-27T22:32:12.940000", "content": "Oof, trying to do this now. GCP is a maze. I get various errors like \"Failed to load: tracking number xxx\", and \"This API cannot be enabled at the moment. You may lack appropriate permissions.\", even though my gcp account is set up. Directions to run jupyter involve running bash scripts, seriously? https://cloud.google.com/dataproc/docs/tutorials/jupyter-notebook? Not sure what's current here and what's obsolete...\nI understand google collab is free, but don't like the 12 hour thing. What do you folks use on GCP? I hate devops.
", "votes": 0, "replies": [] }, { "id": 428911, "author_name": "Ankit Sati", "author_url": "", "post_date": "2018-11-28T04:13:23.217000", "content": "I use fastai installation method. It worked for me in the past so i never bothered to waste precious hours on changing that.
", "votes": 0, "replies": [] } ] }, { "id": 427606, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2018-11-25T20:34:53.727000", "content": "Not doing full data set each epoch, but my final model will have run for almost 150 hours when it stops sometime this evening. This is an older PC but is running with 3 GPU's.
", "votes": 0, "replies": [] }, { "id": 427591, "author_name": "Eric Bouteillon", "author_url": "", "post_date": "2018-11-25T19:43:55.880000", "content": "2 epochs of 7 hours got me lb 935. I guess I need to go deeper or reduce learning rate...
", "votes": 0, "replies": [] }, { "id": 427515, "author_name": "Strideradu", "author_url": "", "post_date": "2018-11-25T17:01:56.153000", "content": "This highly depends on your model and your GPU but I think 1 epoch takes 1-2 days is normal
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "427446": "22 hours for 1 epoch, model converge spend about 4 to 5 days.....", "427430": "Joined this competition late and was wondering what is the average training time for everyone. My model seems to be running for last 23 hours and is at 0.93 CV now.", "428836": "70 hours per epoch for fine tuning on 6*K80.......", "428600": "I've made my data generator such that I can use a subset of classes (for quicker experimentation with a smaller dataset).\n\nhttps://github.com/colinator/doodle_generator/blob/master/data_generator_uniform_final.ipynb\n\nBut, it's revealed to me that I probably can't afford this contest. To run 5 epochs through 100,000 samples of 34 classes, with a mobilenet, takes ~3 hours. So with all 340 classes, it'll take ~30 hours. Paperspace charges $0.78/hour for a nice gpu machine. Wish I had more hardware!", "427606": "Not doing full data set each epoch, but my final model will have run for almost 150 hours when it stops sometime this evening. This is an older PC but is running with 3 GPU's.", "427591": "2 epochs of 7 hours got me lb 935. I guess I need to go deeper or reduce learning rate...", "427515": "This highly depends on your model and your GPU but I think 1 epoch takes 1-2 days is normal" } }