This goes for almost all AI language models that I’ve used.
I ask it a question, and instead of giving me a paragraph instantly, it generates a response word by word, sometimes sticking on a word for a second or two. Why can’t it just paste the entire answer straight away?
In: Technology
It can’t give you a paragraph instantly, because the paragraph is not instantly available.
It is not a rendering gimmick. It is not generating the block of text in one go, and then dripping it out to the recipient purely for the aesthetics. The stream is fundamentally how it is working. It’s a iterative process, and you’re seeing each iteration in real time as each word is being predicted. The models work by taking a body of text as a prompt and then predicting what word should come next*. Each time a new word is generated that new word is added to the prompt, and then that whole new prompt is used in the next iteration. This is what allows successive iterations to remain “aware” of what has been generated thus far.
The UI could have been created so that this whole cycle is allowed to complete before printing the final result, but this would mean waiting for the last word not getting the paragraph instantly. It may as well print each new word as and when it is available. When it gets stuck for a few seconds, it genuinely is waiting for that word to be generated.
*with some randomness to produce variety. It picks from the top candidates within an assigned threshold called the temperature.
Latest Answers