It seems like they all happily make up a completely incorrect answer and never simply say “I don’t know”. It seems like hallucinated answers come when there’s not a lot of information to train them on a topic. Why can’t the model recognize the low amount of training data and generate with a confidence score to determine if they’re making stuff up?
EDIT: Many people point out rightly that the LLMs themselves can’t “understand” their own response and therefore cannot determine if their answers are made up. But I guess the question includes the fact that chat services like ChatGPT already have support services like the Moderation API that evaluate the content of your query and it’s own responses for content moderation purposes, and intervene when the content violates their terms of use. So couldn’t you have another service that evaluates the LLM response for a confidence score to make this work? Perhaps I should have said “LLM chat services” instead of just LLM, but alas, I did not.
In: Technology
Because what they are doing is building sentences via a probability model of what words follow each other most often in a specific context. ChatGPT doesn’t know the meaning of any of the words it strings together and cannot ever know them. ChatGPT even cuts and pastes entire paragraphs from random sources that seem to mathematically match the prompt it was given.
They’re very good at building patterns of words that work together, but any human reading the output can spot the nonsense with even a little research.
Any ‘confidence’ score would rely on understanding meaning, and can therefore only be applied or estimated by humans.
Latest Answers