What about GPU Architecture makes them superior for training neural networks over CPUs?

Question

In ML/AI, GPUs are used to train neural networks of various sizes. They are vastly superior to training on CPUs. Why is this?

In: 679

Answer 1

To give a more high level response:

CPUs are designed to be pretty good at anything, since they have to be able to run any sort of program that a user might want. They’re flexible, at the cost of not being super optimized for any one particular task.

GPUs are designed to be *very* good at a few specific things, mainly the kind of math used to render graphics. They can be very optimized because they only have to do certain tasks. The downside is, they’re not as good at other things.

The kind of math used to render graphics happens to also be the kind of math used in neural networks (mainly linear algebra, which involves processing lots of numbers at once in parallel).

As a matter of fact, companies like Google have now designed even more optimized hardware specifically for neural networks, including Google’s TPUs (tensor processing units; tensors are math objects used in neural nets). Like GPUs, they trade flexibility for being really really good at one thing.

Answer 2

To give a more high level response:

CPUs are designed to be pretty good at anything, since they have to be able to run any sort of program that a user might want. They’re flexible, at the cost of not being super optimized for any one particular task.

GPUs are designed to be *very* good at a few specific things, mainly the kind of math used to render graphics. They can be very optimized because they only have to do certain tasks. The downside is, they’re not as good at other things.

The kind of math used to render graphics happens to also be the kind of math used in neural networks (mainly linear algebra, which involves processing lots of numbers at once in parallel).

As a matter of fact, companies like Google have now designed even more optimized hardware specifically for neural networks, including Google’s TPUs (tensor processing units; tensors are math objects used in neural nets). Like GPUs, they trade flexibility for being really really good at one thing.