Why can’t LLM’s like ChatGPT calculate a confidence score when providing an answer to your question and simply reply “I don’t know” instead of hallucinating an answer?

Question

It seems like they all happily make up a completely incorrect answer and never simply say “I don’t know”. It seems like hallucinated answers come when there’s not a lot of information to train them on a topic. Why can’t the model recognize the low amount of training data and generate with a confidence score to determine if they’re making stuff up?

EDIT: Many people point out rightly that the LLMs themselves can’t “understand” their own response and therefore cannot determine if their answers are made up. But I guess the question includes the fact that chat services like ChatGPT already have support services like the Moderation API that evaluate the content of your query and it’s own responses for content moderation purposes, and intervene when the content violates their terms of use. So couldn’t you have another service that evaluates the LLM response for a confidence score to make this work? Perhaps I should have said “LLM chat services” instead of just LLM, but alas, I did not.

In: Technology

Answer 1

All AI responses are hallucinations. It’s an unfortunate thing that we say that when it gives a crazy answer.

Answer 2

Because as far as the LLM is concerned EVERY answer is a hallucination, the only difference is sometimes that hallucination is correct, and other times it isn’t. 

Answer 3

If it could do that, then it could generate you a better answer …?

Answer 4

A friend who works with LLMs a lot has written a script that asks the LLMs to reconsider their previous answer and give a confidence rating to it, or to answer the question differently X times, and then select from those answers the ones it is most confident in.

It’s a clever bit and seems to help it produce better answers generally.

But still, these are just constructing sentences. They don’t actually understand in a truly intelligent way.

Answer 5

Here’s a fun test for “thinking”. Give ChatGPT or any other LLM bot the following prompt:

>Please write me ten sentences that end with the word “club”. Then review those sentences and see whether they do actually end with the word “club”. Give yourself a score out of 10.

The results are *tragic*.