eli5 hello, I dont understand what tokens are and how they work with chatgpt if someone can explain it to me please.

Question

hello, I dont understand what tokens are and how they work with chatgpt if someone can explain it to me please.

In: Technology

Answer 1

All you need to know is that “Tokens” are basically just “words”.

You can think of it like each word is just its own token.

Internally its not like this, tokens are more nuanced, like ____s could be a token that makes a word plural, and there is a <new paragraph here> token, etc. But unless you are building your own, you can just think of it as “words”