What about GPU Architecture makes them superior for training neural networks over CPUs?

Question

In ML/AI, GPUs are used to train neural networks of various sizes. They are vastly superior to training on CPUs. Why is this?

In: 679

Answer 1

CPUs work on small chunk of data at a time. Many of its instructions rely on the previous one. You can’t solve D=C*3 until you’ve previously solved C=B/A

GPUs work on wide Arrays of data at the same time because that’s what graphics operations are. Here’s a texture, here’s a lighting map, smush them together and draw.

If you have a set of inputs A and weights B that you need to combine together to get output array C then a CPU has to do A[0]+B[0]=C[0] then A[1]+B[1]=C[1] and slowly increment its way through the array with lots of small memory calls

A GPU will take all of A, all of B, split them to how ever many processing nodes are required and solve for all of C in a single instruction step. It’ll take it a bit longer than the CPU can solve A[0]+B[0] but if the array is large then you come out ahead

Since neural networks get better the bigger you make them they end up benefiting from a GPU which can process thousands of weights and values at the same time. For a small neural network a big CPU may be faster because it can process each individual step faster but GPUs win out as soon as you start wanting to do hundreds or thousands of similar equations at the same time

Answer 2

CPUs work on small chunk of data at a time. Many of its instructions rely on the previous one. You can’t solve D=C*3 until you’ve previously solved C=B/A

GPUs work on wide Arrays of data at the same time because that’s what graphics operations are. Here’s a texture, here’s a lighting map, smush them together and draw.

If you have a set of inputs A and weights B that you need to combine together to get output array C then a CPU has to do A[0]+B[0]=C[0] then A[1]+B[1]=C[1] and slowly increment its way through the array with lots of small memory calls

A GPU will take all of A, all of B, split them to how ever many processing nodes are required and solve for all of C in a single instruction step. It’ll take it a bit longer than the CPU can solve A[0]+B[0] but if the array is large then you come out ahead

Since neural networks get better the bigger you make them they end up benefiting from a GPU which can process thousands of weights and values at the same time. For a small neural network a big CPU may be faster because it can process each individual step faster but GPUs win out as soon as you start wanting to do hundreds or thousands of similar equations at the same time