In ML/AI, GPUs are used to train neural networks of various sizes. They are vastly superior to training on CPUs. Why is this?
In: 679
GPUs are optimized for tasks where you need to perform the same operation on thousands of objects at the same time because they usually do very similar calculations for every pixel of the screen. Neural network training gives you more or less this: you need to recalculate parameters for each neuron with mostly the same formula.
CPUs only have a few cores so they would have to recalculate these neurons one by one instead of hundreds at a time, greatly reducing the speed.
Each CPU core tends to have 1 floating point unit, maybe a very small number of arithmetic units, etc. While each CPU core has many operating modes, lots of features, the amount of calculation it can do is more limited as a result. A lot of the CPU’s actual circuitry is dedicated to things other than actual computation, like instruction processing and event ordering.
A GPU’s equivalent of a CPU core has dozens, maybe hundreds, of floating point units available to it. Basically a single instruction can order all floating point units it controls to simultaneously perform the operation `x += y` or such. However each such core is more limited, and anything that can’t make good use of that bulk of FPUs will seriously hurt performance. Furthermore it has generally fewer features available.
GPUs tend to do best when the job involves more calculation and less decision making along the process.
TLDR; A CPU can perform many different tasks but a GPU can perform one task very, very efficiently and that task is computation.
Your CPU is designed to run different kind of tasks. It is general-purpose and flexible: you can play games, listen to music, watch a movie, access websites, all at once. But because of that, it is not the most efficient in doing any of those things.
OTOH GPUs are designed for computational efficiency. (That’s it.) And neural networks are made of repetitve calculations: multiply two numbers and then add another number. You do this for every neuron in the network, for thousands of cycles. For repetitive calculations such as these, GPUs can perform them in parallel on a scale vastly larger than a CPU.
CPUs work on small chunk of data at a time. Many of its instructions rely on the previous one. You can’t solve D=C*3 until you’ve previously solved C=B/A
GPUs work on wide Arrays of data at the same time because that’s what graphics operations are. Here’s a texture, here’s a lighting map, smush them together and draw.
If you have a set of inputs A and weights B that you need to combine together to get output array C then a CPU has to do A[0]+B[0]=C[0] then A[1]+B[1]=C[1] and slowly increment its way through the array with lots of small memory calls
A GPU will take all of A, all of B, split them to how ever many processing nodes are required and solve for all of C in a single instruction step. It’ll take it a bit longer than the CPU can solve A[0]+B[0] but if the array is large then you come out ahead
Since neural networks get better the bigger you make them they end up benefiting from a GPU which can process thousands of weights and values at the same time. For a small neural network a big CPU may be faster because it can process each individual step faster but GPUs win out as soon as you start wanting to do hundreds or thousands of similar equations at the same time
To give a more high level response:
CPUs are designed to be pretty good at anything, since they have to be able to run any sort of program that a user might want. They’re flexible, at the cost of not being super optimized for any one particular task.
GPUs are designed to be *very* good at a few specific things, mainly the kind of math used to render graphics. They can be very optimized because they only have to do certain tasks. The downside is, they’re not as good at other things.
The kind of math used to render graphics happens to also be the kind of math used in neural networks (mainly linear algebra, which involves processing lots of numbers at once in parallel).
As a matter of fact, companies like Google have now designed even more optimized hardware specifically for neural networks, including Google’s TPUs (tensor processing units; tensors are math objects used in neural nets). Like GPUs, they trade flexibility for being really really good at one thing.
One key aspect of GPU architecture that makes them suitable for training neural networks is the presence of many small, efficient processing units, known as “cores,” which can work in parallel to perform the numerous calculations required by machine learning algorithms. This parallel processing capability allows GPUs to perform computations much faster than CPUs, which are designed to handle a single task at a time.
In addition to their parallel processing capabilities, GPUs also have fast memory access and high memory bandwidth, which allows them to efficiently load and process large amounts of data. This is important for machine learning applications, which often require large amounts of data to be processed in order to train and evaluate models.
GPUs have thousands or even 10s of thousands of cores, vs a CPU with single digit or maybe 10s of cores.
GPU cores can only do maths (vs CPU cores that need to handle all kinds of logic), but the difficult part of AI training is loads and loads of maths so a GPU handles that much faster.
GPUs are very good at doing the same thing over and over again on a huge pile of data.
Each pixel in an image (and there may be millions) will have an equation relating it to a texture and then a series of vector or matrix calculations to give a final pixel colour. The same equation is used for every pixel in an object, its just that each pixel has slightly different data (different coordinate).
CPUs are very good at switching from one task to another and hopping about doing different things one after another.
Training neural networks is all about doing the same calculation over and over on a ton of data
In particular it’s mainly matrix operations (or tensor operations, but these can be broken down into matrix operations) which is exactly what GPUs are good at.
Imagine you have 1 million math assignments to do, they are very simple assignments, but there are a lot that need to be done, they are not dependent on each other so they can be done on any order.
You have two options, distribute them to 10 thousand people to do it in parallel or give them to 10 math experts. The experts are very fast, but hey, there are only 10 of them, the 10 thousand are more suitable for the task because they have the “brute force” for this.
GPUs have thousands of cores, CPUs have tens.
A lot of what machine learning does is multiplying vectors, which so happens to be what GPU’s are designed to do as well. GPU’s do it to calculate with polygons. And most of what machine learning does is as said multiplying vectors which makes them a great fit.
Not to mention that a good CPU has at the very top end 64 cores wheras a GPU has thousnads of compute units and also a far wider data bus.
In ML/AI, GPUs are used to train neural networks of various sizes. They are vastly superior to training on CPUs. Why is this?
In: 679
GPUs are optimized for tasks where you need to perform the same operation on thousands of objects at the same time because they usually do very similar calculations for every pixel of the screen. Neural network training gives you more or less this: you need to recalculate parameters for each neuron with mostly the same formula.
CPUs only have a few cores so they would have to recalculate these neurons one by one instead of hundreds at a time, greatly reducing the speed.
Each CPU core tends to have 1 floating point unit, maybe a very small number of arithmetic units, etc. While each CPU core has many operating modes, lots of features, the amount of calculation it can do is more limited as a result. A lot of the CPU’s actual circuitry is dedicated to things other than actual computation, like instruction processing and event ordering.
A GPU’s equivalent of a CPU core has dozens, maybe hundreds, of floating point units available to it. Basically a single instruction can order all floating point units it controls to simultaneously perform the operation `x += y` or such. However each such core is more limited, and anything that can’t make good use of that bulk of FPUs will seriously hurt performance. Furthermore it has generally fewer features available.
GPUs tend to do best when the job involves more calculation and less decision making along the process.
TLDR; A CPU can perform many different tasks but a GPU can perform one task very, very efficiently and that task is computation.
Your CPU is designed to run different kind of tasks. It is general-purpose and flexible: you can play games, listen to music, watch a movie, access websites, all at once. But because of that, it is not the most efficient in doing any of those things.
OTOH GPUs are designed for computational efficiency. (That’s it.) And neural networks are made of repetitve calculations: multiply two numbers and then add another number. You do this for every neuron in the network, for thousands of cycles. For repetitive calculations such as these, GPUs can perform them in parallel on a scale vastly larger than a CPU.
CPUs work on small chunk of data at a time. Many of its instructions rely on the previous one. You can’t solve D=C*3 until you’ve previously solved C=B/A
GPUs work on wide Arrays of data at the same time because that’s what graphics operations are. Here’s a texture, here’s a lighting map, smush them together and draw.
If you have a set of inputs A and weights B that you need to combine together to get output array C then a CPU has to do A[0]+B[0]=C[0] then A[1]+B[1]=C[1] and slowly increment its way through the array with lots of small memory calls
A GPU will take all of A, all of B, split them to how ever many processing nodes are required and solve for all of C in a single instruction step. It’ll take it a bit longer than the CPU can solve A[0]+B[0] but if the array is large then you come out ahead
Since neural networks get better the bigger you make them they end up benefiting from a GPU which can process thousands of weights and values at the same time. For a small neural network a big CPU may be faster because it can process each individual step faster but GPUs win out as soon as you start wanting to do hundreds or thousands of similar equations at the same time
To give a more high level response:
CPUs are designed to be pretty good at anything, since they have to be able to run any sort of program that a user might want. They’re flexible, at the cost of not being super optimized for any one particular task.
GPUs are designed to be *very* good at a few specific things, mainly the kind of math used to render graphics. They can be very optimized because they only have to do certain tasks. The downside is, they’re not as good at other things.
The kind of math used to render graphics happens to also be the kind of math used in neural networks (mainly linear algebra, which involves processing lots of numbers at once in parallel).
As a matter of fact, companies like Google have now designed even more optimized hardware specifically for neural networks, including Google’s TPUs (tensor processing units; tensors are math objects used in neural nets). Like GPUs, they trade flexibility for being really really good at one thing.
One key aspect of GPU architecture that makes them suitable for training neural networks is the presence of many small, efficient processing units, known as “cores,” which can work in parallel to perform the numerous calculations required by machine learning algorithms. This parallel processing capability allows GPUs to perform computations much faster than CPUs, which are designed to handle a single task at a time.
In addition to their parallel processing capabilities, GPUs also have fast memory access and high memory bandwidth, which allows them to efficiently load and process large amounts of data. This is important for machine learning applications, which often require large amounts of data to be processed in order to train and evaluate models.
GPUs have thousands or even 10s of thousands of cores, vs a CPU with single digit or maybe 10s of cores.
GPU cores can only do maths (vs CPU cores that need to handle all kinds of logic), but the difficult part of AI training is loads and loads of maths so a GPU handles that much faster.
GPUs are very good at doing the same thing over and over again on a huge pile of data.
Each pixel in an image (and there may be millions) will have an equation relating it to a texture and then a series of vector or matrix calculations to give a final pixel colour. The same equation is used for every pixel in an object, its just that each pixel has slightly different data (different coordinate).
CPUs are very good at switching from one task to another and hopping about doing different things one after another.
Training neural networks is all about doing the same calculation over and over on a ton of data
In particular it’s mainly matrix operations (or tensor operations, but these can be broken down into matrix operations) which is exactly what GPUs are good at.
Imagine you have 1 million math assignments to do, they are very simple assignments, but there are a lot that need to be done, they are not dependent on each other so they can be done on any order.
You have two options, distribute them to 10 thousand people to do it in parallel or give them to 10 math experts. The experts are very fast, but hey, there are only 10 of them, the 10 thousand are more suitable for the task because they have the “brute force” for this.
GPUs have thousands of cores, CPUs have tens.
A lot of what machine learning does is multiplying vectors, which so happens to be what GPU’s are designed to do as well. GPU’s do it to calculate with polygons. And most of what machine learning does is as said multiplying vectors which makes them a great fit.
Not to mention that a good CPU has at the very top end 64 cores wheras a GPU has thousnads of compute units and also a far wider data bus.
Latest Answers