I’ve seen all the examples of the crazy stuff ChatGPT can do, but the one that really gets me is the coding. It can explain what a function does, and write novel syntactically correct code. How on *earth* can it do that? What is in its training set, and how does it work out the meaning behind code?
In: 3
I would guess the developers have trained it on large datasets, scrapped or contributed from a combination of volunteers, internet archives, research projects and other similar sources. Large companies, like Google for example, have spent millions (if not more) for years on this type of stuff.
Here’s something random I tried: https://imgur.com/a/fZL95Q1
Fascinating!
GPT, and similar systems are, at their heart, about performing matrix operations on vectors. Your text prompt is converted to a numerical vector, GPT applies its matrices (that have been fine tuned on training data), then the resulting vector is converted back to text. It doesn’t conceptualise or reason about its inputs in any human-like way. It can produce such human-like output either by repeating snippets of its training data (this would be considered overfitting, and the developers would try to minimise it), or by generalising from common trends in the training data.
Another commenter made the point that it likely learned from documentation and code discussion fora. It may also be that this type of text generalises particularly well.
Latest Answers