Hi everyone,
\n\nI am planning to build a deep learning rig. But I don't have much idea about the compatibilities of various components. Can you please post the specifications of your machines like: whicj processor or motherboard you are using and how much was the total cost.
\n\nThanks.
", "messages": [ { "id": 419452, "postDate": "2018-11-12T01:49:11.983Z", "content": "Mukesh
\n\nIt's all about the money! How much can you spend on hardware (ie cash in hand) vs monthly cloud payments.
\n\nI am a real Python rookie - so much of what I do consumes time without real progress on getting to a good model. I have two rigs with multiple GPU's. One has two 11GB GPU's and the second has three 8GB GPU's. Sadly I am a Windows master and a Linux flunk out. So both rigs on Windows 10. Windows 10 takes 20%+ of the GPU memory and will not let go - so my 11GB rig drops to 9GB as soon as I turn it on, and the 8 goes to 6. Over the years I have enjoyed breaking and fixing PC's, so doing this is part of my fun.
\n\nI pretty much have both of them running 24/7 on one or more of the challenges. No way I want those cloud payments.
\n\nDecision point 1 - if you know and love Linux than that's a vote for your own rig.
\n\nTensorflow is pretty easy to get going with multiple GPU's. Only tricky part is saving models and reloading. I have not explored other frameworks. Starting to try and run XGBoost over the next weeks but the GPU version looks messy. As noted by remidi - its not always easy to get mutliple GPU's running. I have my 3 GPU rig on Windows 10 fast track - about every month or so a Windows update kills me. But I am in this to learn - learning to run Python is as big a piece of the pie as learning to code. So I embrace the hardware crashes.
\n\nDecision point 2 - going to run tensorflow most of the time - easy to do. Cannot speak to easy of multiple GPU on anything else.
\n\nCPU's - I used what I had. The 3 rig is an older machine. The two rig is recent build - both were Intel and the best I could afford at the time ($400 each). I tried a third generation older machine - probably a decade in age - the CPU could not feed a single GPU, let along two or three. I got both machines to a two GPU level by just adding the cards. At full overclock both machines die - they run hot and high watts. I have to run them at around 90% to be stable. Two cards just barely fit. To get to three GPU I bought a 40 dollar mining frame and all three GPU external to the PC case, with lots of cooling air. As noted by Buluga this makes for a very noisy work environment. I don't sleep close, but the TV volume is up a notch or two when I am watching. I plan to move the noisy guy to the basement - I use Multiplicity for KMV.
\n\nDecision point 3 - your current PC - if a desktop with recent I7 level CPU than do it - less CPU power or more age than its shaky. Got only a laptop - hmmm you really need to love deep learning to make that plunge.
\n\nGPU's - now we are talking money. Aside from a $40 mining frame , a box of risers (also $40) and some cables all the money so far goes to buying GPU's. My three rig is using $500 GPU - to get to 4 GPU I need a new motherboard, etc. My two rig is using $1000 GPU - to go from 2 to 3 I need the external framework - I can get 3 in the current case but you could bake a turkey for thanksgiving from the trapped heat. If I am still loving Kaggle after Christmas Santa will be bringing me a third GPU.
\n\nA rig with single GPU can be converted to any other task you might enjoy in the future. Maybe gaming can benefit from a dual GPU rig, and mining bitcoins would be the only future use for a multiple GPU rig. If your a serious gamer, than another decision point that would vote to build the rig.
\n\nIf your starting from scratch with the need for a clean build than I am not much help. Next year I will be building a rig from scratch (if still doing this Python thing) - have spent some time looking but no firm specs. All the cyberbucks mining rigs are built with low power CPU - the GPU's do all the hash work. I will likely try an expensive mining motherboard with the best Intel chip it can run. Lots of mining rigs for sale on Ebay, but all priced pretty much on the number and quality of the GPU and come with minimum memory and low level CPU - so I am looking for a bargain sale where I can drive and pick it up (shipping costs a bunch) and upgrade CPU and memory as needed to get to a 6 or more rig for deep learning.
", "rawMarkdown": "Mukesh\n\nIt's all about the money! How much can you spend on hardware (ie cash in hand) vs monthly cloud payments.\n\nI am a real Python rookie - so much of what I do consumes time without real progress on getting to a good model. I have two rigs with multiple GPU's. One has two 11GB GPU's and the second has three 8GB GPU's. Sadly I am a Windows master and a Linux flunk out. So both rigs on Windows 10. Windows 10 takes 20%+ of the GPU memory and will not let go - so my 11GB rig drops to 9GB as soon as I turn it on, and the 8 goes to 6. Over the years I have enjoyed breaking and fixing PC's, so doing this is part of my fun.\n\nI pretty much have both of them running 24/7 on one or more of the challenges. No way I want those cloud payments.\n\nDecision point 1 - if you know and love Linux than that's a vote for your own rig.\n\nTensorflow is pretty easy to get going with multiple GPU's. Only tricky part is saving models and reloading. I have not explored other frameworks. Starting to try and run XGBoost over the next weeks but the GPU version looks messy. As noted by remidi - its not always easy to get mutliple GPU's running. I have my 3 GPU rig on Windows 10 fast track - about every month or so a Windows update kills me. But I am in this to learn - learning to run Python is as big a piece of the pie as learning to code. So I embrace the hardware crashes. \n\nDecision point 2 - going to run tensorflow most of the time - easy to do. Cannot speak to easy of multiple GPU on anything else.\n\nCPU's - I used what I had. The 3 rig is an older machine. The two rig is recent build - both were Intel and the best I could afford at the time ($400 each). I tried a third generation older machine - probably a decade in age - the CPU could not feed a single GPU, let along two or three. I got both machines to a two GPU level by just adding the cards. At full overclock both machines die - they run hot and high watts. I have to run them at around 90% to be stable. Two cards just barely fit. To get to three GPU I bought a 40 dollar mining frame and all three GPU external to the PC case, with lots of cooling air. As noted by Buluga this makes for a very noisy work environment. I don't sleep close, but the TV volume is up a notch or two when I am watching. I plan to move the noisy guy to the basement - I use Multiplicity for KMV.\n\nDecision point 3 - your current PC - if a desktop with recent I7 level CPU than do it - less CPU power or more age than its shaky. Got only a laptop - hmmm you really need to love deep learning to make that plunge.\n\nGPU's - now we are talking money. Aside from a $40 mining frame , a box of risers (also $40) and some cables all the money so far goes to buying GPU's. My three rig is using $500 GPU - to get to 4 GPU I need a new motherboard, etc. My two rig is using $1000 GPU - to go from 2 to 3 I need the external framework - I can get 3 in the current case but you could bake a turkey for thanksgiving from the trapped heat. If I am still loving Kaggle after Christmas Santa will be bringing me a third GPU.\n\nA rig with single GPU can be converted to any other task you might enjoy in the future. Maybe gaming can benefit from a dual GPU rig, and mining bitcoins would be the only future use for a multiple GPU rig. If your a serious gamer, than another decision point that would vote to build the rig.\n\nIf your starting from scratch with the need for a clean build than I am not much help. Next year I will be building a rig from scratch (if still doing this Python thing) - have spent some time looking but no firm specs. All the cyberbucks mining rigs are built with low power CPU - the GPU's do all the hash work. I will likely try an expensive mining motherboard with the best Intel chip it can run. Lots of mining rigs for sale on Ebay, but all priced pretty much on the number and quality of the GPU and come with minimum memory and low level CPU - so I am looking for a bargain sale where I can drive and pick it up (shipping costs a bunch) and upgrade CPU and memory as needed to get to a 6 or more rig for deep learning.\n\n", "votes": 5 }, { "id": 408024, "postDate": "2018-10-22T07:24:51.537Z", "content": "Great links @hendraherviawan. I agree that deep learning is not only about tensors flowing through the fitted model, but also about the tensors flowing through the hardware setup.
", "rawMarkdown": "Great links @hendraherviawan. I agree that deep learning is not only about tensors flowing through the fitted model, but also about the tensors flowing through the hardware setup. " } ] }, { "id": 403470, "postDate": "2018-10-13T17:26:57.143Z", "content": "Hi everyone,
\n\nI am planning to build a deep learning rig. But I don't have much idea about the compatibilities of various components. Can you please post the specifications of your machines like: whicj processor or motherboard you are using and how much was the total cost.
\n\nThanks.
", "rawMarkdown": "Hi everyone, \n\nI am planning to build a deep learning rig. But I don't have much idea about the compatibilities of various components. Can you please post the specifications of your machines like: whicj processor or motherboard you are using and how much was the total cost.\n\nThanks.", "votes": -4 }, { "id": 410016, "postDate": "2018-10-25T08:02:27.197Z", "content": "I do understand your requirement but being a owner of hardware rig, 1080ti. Please don't build it. It is so much of maintenance and very less of data science work. I did build my own machine and currently working on it but the os, nvidia drivers, cuda, and ml/dl library versions keep changing and the compatibility is not so good and it becomes hard to update.\n It is better to use cloud where all of this comes pre-built and you can concentrate on building the data science product.
", "rawMarkdown": "I do understand your requirement but being a owner of hardware rig, 1080ti. Please don't build it. It is so much of maintenance and very less of data science work. I did build my own machine and currently working on it but the os, nvidia drivers, cuda, and ml/dl library versions keep changing and the compatibility is not so good and it becomes hard to update.\n It is better to use cloud where all of this comes pre-built and you can concentrate on building the data science product.", "votes": -3, "replies": [ { "id": 410049, "postDate": "2018-10-25T09:23:03.043Z", "content": "I totally have to disagree here. For me setup of a new machine is like 1 hour and I never experienced compability problems.
", "rawMarkdown": "I totally have to disagree here. For me setup of a new machine is like 1 hour and I never experienced compability problems.", "votes": 2 }, { "id": 410099, "postDate": "2018-10-25T12:04:24.540Z", "content": "Agree with joseph, Conda now will otomatically download cuda & cudnn as dependency if you install tensorflow-gpu
\n\nconda create --name tf_gpu tensorflow-gpu
Hey Mukesh,\nI personally use Google Cloud I like the flexibility and the economics of it. Have you done any research in terms of pricing if its cheaper to build rig or use AWS or gcloud ?
", "rawMarkdown": "Hey Mukesh,\nI personally use Google Cloud I like the flexibility and the economics of it. Have you done any research in terms of pricing if its cheaper to build rig or use AWS or gcloud ?", "replies": [ { "id": 405027, "postDate": "2018-10-16T18:46:59.483Z", "content": "I don't know about the cost of Clouds. Can you tell how much it costs for one computer vision kaggle competition?
", "rawMarkdown": "I don't know about the cost of Clouds. Can you tell how much it costs for one computer vision kaggle competition?" }, { "id": 405076, "postDate": "2018-10-16T20:37:24.827Z", "content": "Even if I am aiming for prize winning spots I usually spend less than $200 for a competition. GCP gives $300 credit to try it imho it should be enough. I haven't upgraded my PC since 2013 and I don't plan to buy new rig I prefer the above mentioned flexibility.
\n\nAnother aspect that GPUs/fans could be noisy and I like to run training while I am sleeping :)
", "rawMarkdown": "Even if I am aiming for prize winning spots I usually spend less than $200 for a competition. GCP gives $300 credit to try it imho it should be enough. I haven't upgraded my PC since 2013 and I don't plan to buy new rig I prefer the above mentioned flexibility.\n\nAnother aspect that GPUs/fans could be noisy and I like to run training while I am sleeping :)" } ] } ], "comments": [ { "id": 419452, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2018-11-12T01:49:11.983000", "content": "Mukesh
\n\nIt's all about the money! How much can you spend on hardware (ie cash in hand) vs monthly cloud payments.
\n\nI am a real Python rookie - so much of what I do consumes time without real progress on getting to a good model. I have two rigs with multiple GPU's. One has two 11GB GPU's and the second has three 8GB GPU's. Sadly I am a Windows master and a Linux flunk out. So both rigs on Windows 10. Windows 10 takes 20%+ of the GPU memory and will not let go - so my 11GB rig drops to 9GB as soon as I turn it on, and the 8 goes to 6. Over the years I have enjoyed breaking and fixing PC's, so doing this is part of my fun.
\n\nI pretty much have both of them running 24/7 on one or more of the challenges. No way I want those cloud payments.
\n\nDecision point 1 - if you know and love Linux than that's a vote for your own rig.
\n\nTensorflow is pretty easy to get going with multiple GPU's. Only tricky part is saving models and reloading. I have not explored other frameworks. Starting to try and run XGBoost over the next weeks but the GPU version looks messy. As noted by remidi - its not always easy to get mutliple GPU's running. I have my 3 GPU rig on Windows 10 fast track - about every month or so a Windows update kills me. But I am in this to learn - learning to run Python is as big a piece of the pie as learning to code. So I embrace the hardware crashes.
\n\nDecision point 2 - going to run tensorflow most of the time - easy to do. Cannot speak to easy of multiple GPU on anything else.
\n\nCPU's - I used what I had. The 3 rig is an older machine. The two rig is recent build - both were Intel and the best I could afford at the time ($400 each). I tried a third generation older machine - probably a decade in age - the CPU could not feed a single GPU, let along two or three. I got both machines to a two GPU level by just adding the cards. At full overclock both machines die - they run hot and high watts. I have to run them at around 90% to be stable. Two cards just barely fit. To get to three GPU I bought a 40 dollar mining frame and all three GPU external to the PC case, with lots of cooling air. As noted by Buluga this makes for a very noisy work environment. I don't sleep close, but the TV volume is up a notch or two when I am watching. I plan to move the noisy guy to the basement - I use Multiplicity for KMV.
\n\nDecision point 3 - your current PC - if a desktop with recent I7 level CPU than do it - less CPU power or more age than its shaky. Got only a laptop - hmmm you really need to love deep learning to make that plunge.
\n\nGPU's - now we are talking money. Aside from a $40 mining frame , a box of risers (also $40) and some cables all the money so far goes to buying GPU's. My three rig is using $500 GPU - to get to 4 GPU I need a new motherboard, etc. My two rig is using $1000 GPU - to go from 2 to 3 I need the external framework - I can get 3 in the current case but you could bake a turkey for thanksgiving from the trapped heat. If I am still loving Kaggle after Christmas Santa will be bringing me a third GPU.
\n\nA rig with single GPU can be converted to any other task you might enjoy in the future. Maybe gaming can benefit from a dual GPU rig, and mining bitcoins would be the only future use for a multiple GPU rig. If your a serious gamer, than another decision point that would vote to build the rig.
\n\nIf your starting from scratch with the need for a clean build than I am not much help. Next year I will be building a rig from scratch (if still doing this Python thing) - have spent some time looking but no firm specs. All the cyberbucks mining rigs are built with low power CPU - the GPU's do all the hash work. I will likely try an expensive mining motherboard with the best Intel chip it can run. Lots of mining rigs for sale on Ebay, but all priced pretty much on the number and quality of the GPU and come with minimum memory and low level CPU - so I am looking for a bargain sale where I can drive and pick it up (shipping costs a bunch) and upgrade CPU and memory as needed to get to a 6 or more rig for deep learning.
", "votes": 5, "replies": [] }, { "id": 408024, "author_name": "M Hendra Herviawan", "author_url": "", "post_date": "2018-10-22T07:24:51.537000", "content": "Great links @hendraherviawan. I agree that deep learning is not only about tensors flowing through the fitted model, but also about the tensors flowing through the hardware setup.
", "votes": 0, "replies": [] } ] }, { "id": 410016, "author_name": "remidi", "author_url": "", "post_date": "2018-10-25T08:02:27.197000", "content": "I do understand your requirement but being a owner of hardware rig, 1080ti. Please don't build it. It is so much of maintenance and very less of data science work. I did build my own machine and currently working on it but the os, nvidia drivers, cuda, and ml/dl library versions keep changing and the compatibility is not so good and it becomes hard to update.\n It is better to use cloud where all of this comes pre-built and you can concentrate on building the data science product.
", "votes": -3, "replies": [ { "id": 410049, "author_name": "Tim Joseph", "author_url": "", "post_date": "2018-10-25T09:23:03.043000", "content": "I totally have to disagree here. For me setup of a new machine is like 1 hour and I never experienced compability problems.
", "votes": 2, "replies": [] }, { "id": 410099, "author_name": "M Hendra Herviawan", "author_url": "", "post_date": "2018-10-25T12:04:24.540000", "content": "Agree with joseph, Conda now will otomatically download cuda & cudnn as dependency if you install tensorflow-gpu
\n\nconda create --name tf_gpu tensorflow-gpu
Hey Mukesh,\nI personally use Google Cloud I like the flexibility and the economics of it. Have you done any research in terms of pricing if its cheaper to build rig or use AWS or gcloud ?
", "votes": 0, "replies": [ { "id": 405027, "author_name": "Mukesh", "author_url": "", "post_date": "2018-10-16T18:46:59.483000", "content": "I don't know about the cost of Clouds. Can you tell how much it costs for one computer vision kaggle competition?
", "votes": 0, "replies": [] }, { "id": 405076, "author_name": "beluga", "author_url": "", "post_date": "2018-10-16T20:37:24.827000", "content": "Even if I am aiming for prize winning spots I usually spend less than $200 for a competition. GCP gives $300 credit to try it imho it should be enough. I haven't upgraded my PC since 2013 and I don't plan to buy new rig I prefer the above mentioned flexibility.
\n\nAnother aspect that GPUs/fans could be noisy and I like to run training while I am sleeping :)
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "419452": "Mukesh\n\nIt's all about the money! How much can you spend on hardware (ie cash in hand) vs monthly cloud payments.\n\nI am a real Python rookie - so much of what I do consumes time without real progress on getting to a good model. I have two rigs with multiple GPU's. One has two 11GB GPU's and the second has three 8GB GPU's. Sadly I am a Windows master and a Linux flunk out. So both rigs on Windows 10. Windows 10 takes 20%+ of the GPU memory and will not let go - so my 11GB rig drops to 9GB as soon as I turn it on, and the 8 goes to 6. Over the years I have enjoyed breaking and fixing PC's, so doing this is part of my fun.\n\nI pretty much have both of them running 24/7 on one or more of the challenges. No way I want those cloud payments.\n\nDecision point 1 - if you know and love Linux than that's a vote for your own rig.\n\nTensorflow is pretty easy to get going with multiple GPU's. Only tricky part is saving models and reloading. I have not explored other frameworks. Starting to try and run XGBoost over the next weeks but the GPU version looks messy. As noted by remidi - its not always easy to get mutliple GPU's running. I have my 3 GPU rig on Windows 10 fast track - about every month or so a Windows update kills me. But I am in this to learn - learning to run Python is as big a piece of the pie as learning to code. So I embrace the hardware crashes. \n\nDecision point 2 - going to run tensorflow most of the time - easy to do. Cannot speak to easy of multiple GPU on anything else.\n\nCPU's - I used what I had. The 3 rig is an older machine. The two rig is recent build - both were Intel and the best I could afford at the time ($400 each). I tried a third generation older machine - probably a decade in age - the CPU could not feed a single GPU, let along two or three. I got both machines to a two GPU level by just adding the cards. At full overclock both machines die - they run hot and high watts. I have to run them at around 90% to be stable. Two cards just barely fit. To get to three GPU I bought a 40 dollar mining frame and all three GPU external to the PC case, with lots of cooling air. As noted by Buluga this makes for a very noisy work environment. I don't sleep close, but the TV volume is up a notch or two when I am watching. I plan to move the noisy guy to the basement - I use Multiplicity for KMV.\n\nDecision point 3 - your current PC - if a desktop with recent I7 level CPU than do it - less CPU power or more age than its shaky. Got only a laptop - hmmm you really need to love deep learning to make that plunge.\n\nGPU's - now we are talking money. Aside from a $40 mining frame , a box of risers (also $40) and some cables all the money so far goes to buying GPU's. My three rig is using $500 GPU - to get to 4 GPU I need a new motherboard, etc. My two rig is using $1000 GPU - to go from 2 to 3 I need the external framework - I can get 3 in the current case but you could bake a turkey for thanksgiving from the trapped heat. If I am still loving Kaggle after Christmas Santa will be bringing me a third GPU.\n\nA rig with single GPU can be converted to any other task you might enjoy in the future. Maybe gaming can benefit from a dual GPU rig, and mining bitcoins would be the only future use for a multiple GPU rig. If your a serious gamer, than another decision point that would vote to build the rig.\n\nIf your starting from scratch with the need for a clean build than I am not much help. Next year I will be building a rig from scratch (if still doing this Python thing) - have spent some time looking but no firm specs. All the cyberbucks mining rigs are built with low power CPU - the GPU's do all the hash work. I will likely try an expensive mining motherboard with the best Intel chip it can run. Lots of mining rigs for sale on Ebay, but all priced pretty much on the number and quality of the GPU and come with minimum memory and low level CPU - so I am looking for a bargain sale where I can drive and pick it up (shipping costs a bunch) and upgrade CPU and memory as needed to get to a 6 or more rig for deep learning.\n\n", "408024": " 1. https://blog.slavv.com/the-1700-great-deep-learning-box-assembly-setup-and-benchmarks-148c5ebe6415\n 1. https://forums.fast.ai/t/its-alive-my-deep-learning-rig-for-part-1/17149\n 1. http://timdettmers.com/2018/08/21/which-gpu-for-deep-learning/\n\n", "403470": "Hi everyone, \n\nI am planning to build a deep learning rig. But I don't have much idea about the compatibilities of various components. Can you please post the specifications of your machines like: whicj processor or motherboard you are using and how much was the total cost.\n\nThanks.", "410016": "I do understand your requirement but being a owner of hardware rig, 1080ti. Please don't build it. It is so much of maintenance and very less of data science work. I did build my own machine and currently working on it but the os, nvidia drivers, cuda, and ml/dl library versions keep changing and the compatibility is not so good and it becomes hard to update.\n It is better to use cloud where all of this comes pre-built and you can concentrate on building the data science product.", "403473": "Hey Mukesh,\nI personally use Google Cloud I like the flexibility and the economics of it. Have you done any research in terms of pricing if its cheaper to build rig or use AWS or gcloud ?" } }