I was trying to train MobileNet on multi gpus using pytorch. From watch nvidia-smi, I see that GPUs are sometimes working, and sometimes not working(GPU util is 0%). This slows down training speed a lot.
But training MobileNet on a single GPU and training ResNet50 on multi GPUs do not have such issue. I was wondering what is going wrong. Is there someone used to meet this problem?
\n\nPS: I read all training data into memory. Pytorch version is 0.4.0
", "messages": [ { "id": 424622, "postDate": "2018-11-20T12:48:39.567Z", "content": "I was trying to train MobileNet on multi gpus using pytorch. From watch nvidia-smi, I see that GPUs are sometimes working, and sometimes not working(GPU util is 0%). This slows down training speed a lot.
But training MobileNet on a single GPU and training ResNet50 on multi GPUs do not have such issue. I was wondering what is going wrong. Is there someone used to meet this problem?
\n\nPS: I read all training data into memory. Pytorch version is 0.4.0
", "rawMarkdown": "I was trying to train MobileNet on multi gpus using pytorch. From `watch nvidia-smi`, I see that GPUs are sometimes working, and sometimes not working(GPU util is 0%). This slows down training speed a lot.\n\nBut training MobileNet on a single GPU and training ResNet50 on multi GPUs do not have such issue. I was wondering what is going wrong. Is there someone used to meet this problem?\n\nPS: I read all training data into memory. Pytorch version is 0.4.0", "votes": 1 }, { "id": 426511, "postDate": "2018-11-23T11:32:33.557Z", "content": "Was facing the same problem. I was using 8 V100s and time for one epoch was nearly same as 4 V100s. I was using data loaders and had to increasse my num_workers(V100s are very fast), after that still usage was dropping to zero.\nI decreased from 8 to 4. Now utilization is alteast not zero at any time. \nSince you are loading all of data into memory, so my solution won't help you. Are you monitoring cpu usage?? The weights are updated on the cpu, try debuggin there.
", "rawMarkdown": "Was facing the same problem. I was using 8 V100s and time for one epoch was nearly same as 4 V100s. I was using data loaders and had to increasse my num_workers(V100s are very fast), after that still usage was dropping to zero.\nI decreased from 8 to 4. Now utilization is alteast not zero at any time. \nSince you are loading all of data into memory, so my solution won't help you. Are you monitoring cpu usage?? The weights are updated on the cpu, try debuggin there." }, { "id": 424923, "postDate": "2018-11-20T22:15:39.423Z", "content": "Do most of my stuff on two or three GPU local PC. Have occasionally seen what your describing. When its extreme two GPU's end up slower than one. I am pretty much a noobie with Python, so I seldom can fix this issue. To get maximum GPU usage I have success with things that increase the CPU usage.\nThings that seem to work.\n1. Larger batch size - make it just short of the size that creates OOM. This seems to do the best job when CPU usage is low.\n2. Make validation size smaller.\n3. Use generator.
\n4. Play with max_q_size callback.
Thanks for you reply. I think you are using Keras. I have also tried Keras, using multi gpus is slower, but in Pytorch, using multi gpus is much more slower than Keras.
", "rawMarkdown": "Thanks for you reply. I think you are using Keras. I have also tried Keras, using multi gpus is slower, but in Pytorch, using multi gpus is much more slower than Keras." } ] } ], "comments": [ { "id": 426511, "author_name": "VikasSangwan", "author_url": "", "post_date": "2018-11-23T11:32:33.557000", "content": "Was facing the same problem. I was using 8 V100s and time for one epoch was nearly same as 4 V100s. I was using data loaders and had to increasse my num_workers(V100s are very fast), after that still usage was dropping to zero.\nI decreased from 8 to 4. Now utilization is alteast not zero at any time. \nSince you are loading all of data into memory, so my solution won't help you. Are you monitoring cpu usage?? The weights are updated on the cpu, try debuggin there.
", "votes": 0, "replies": [] }, { "id": 424923, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2018-11-20T22:15:39.423000", "content": "Do most of my stuff on two or three GPU local PC. Have occasionally seen what your describing. When its extreme two GPU's end up slower than one. I am pretty much a noobie with Python, so I seldom can fix this issue. To get maximum GPU usage I have success with things that increase the CPU usage.\nThings that seem to work.\n1. Larger batch size - make it just short of the size that creates OOM. This seems to do the best job when CPU usage is low.\n2. Make validation size smaller.\n3. Use generator.
\n4. Play with max_q_size callback.
Thanks for you reply. I think you are using Keras. I have also tried Keras, using multi gpus is slower, but in Pytorch, using multi gpus is much more slower than Keras.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "424622": "I was trying to train MobileNet on multi gpus using pytorch. From `watch nvidia-smi`, I see that GPUs are sometimes working, and sometimes not working(GPU util is 0%). This slows down training speed a lot.\n\nBut training MobileNet on a single GPU and training ResNet50 on multi GPUs do not have such issue. I was wondering what is going wrong. Is there someone used to meet this problem?\n\nPS: I read all training data into memory. Pytorch version is 0.4.0", "426511": "Was facing the same problem. I was using 8 V100s and time for one epoch was nearly same as 4 V100s. I was using data loaders and had to increasse my num_workers(V100s are very fast), after that still usage was dropping to zero.\nI decreased from 8 to 4. Now utilization is alteast not zero at any time. \nSince you are loading all of data into memory, so my solution won't help you. Are you monitoring cpu usage?? The weights are updated on the cpu, try debuggin there.", "424923": "Do most of my stuff on two or three GPU local PC. Have occasionally seen what your describing. When its extreme two GPU's end up slower than one. I am pretty much a noobie with Python, so I seldom can fix this issue. To get maximum GPU usage I have success with things that increase the CPU usage.\nThings that seem to work.\n1. Larger batch size - make it just short of the size that creates OOM. This seems to do the best job when CPU usage is low.\n2. Make validation size smaller.\n3. Use generator. \n4. Play with max_q_size callback." } }