Why does AI provide a confident wrong answer more often than an honest "I don't know"?

Why does AI provide a confident wrong answer more often than an honest "I don't know"?

Neural networks love benchmarks where they win by a landslide. But as soon as researchers devised a test that penalizes not for ignorance, but for confident lying, the picture changed dramatically. Most top models failed—and they failed predictably, in the exact same way.

A test that punishes guessing

We are talking about the AA-Omniscience Index from Artificial Analysis—a benchmark consisting of thousands of questions across business, law, medicine, science, and development. Its mechanics are simple and fair: it is a bounded metric from -100 to 100 that measures factual memory, simultaneously penalizing hallucinations and rewarding the refusal to answer when uncertain, where 0 corresponds to a model that answers questions correctly as often as it does incorrectly. In other words, a model can honestly say "I don't know" without a penalty, but making up a beautiful yet incorrect answer costs it points.

The result is sobering: out of 40 tested models, only four received a positive score. The rest were wrong more often than they were right when a question went beyond the scope of their reliable knowledge.

The higher the accuracy, the greater the temptation to lie

The most interesting part is the connection between "knowing a lot" and "not lying." One would think they should go hand in hand. In practice, it is the opposite: good knowledge does not guarantee low hallucination—the hallucination metric measures how often a model guesses when it lacks the necessary knowledge, and the models with the highest accuracy do not lead the index precisely because of their tendency to guess instead of abstaining.

The mechanics are banally simple: for every complex question, a model has three paths—answer correctly, answer incorrectly, or honestly refuse. When an AI receives a question, it does one of three things: answers correctly, answers incorrectly, or refuses to answer—these three options total 100%. Models tuned to "always answer" simply shift their lack of knowledge into the "incorrect" column rather than the "refusal" column.

Some models chose a different strategy—refusing more often. Due to Anthropic's approach to Constitutional AI training, Claude is more inclined to refuse than to guess when it is unsure. This does not make the model "smarter" in terms of raw accuracy, but it sharply reduces the number of confident fabrications—which is often more important for practical tasks.

An error that isn't actually an error

There is also the flip side—when a model is formally "wrong" but acts more reasonably than the test itself. One example emerged during the testing of Claude Opus 4.5 on an agentic benchmark: the model solved a case about an airline in an unconventional way, and it was counted as a failure. The benchmark formally marked it as a failure because the method Claude used to help the client was unforeseen. But it is precisely this creative problem-solving that makes a model significantly more advanced. At the same time, different models behaved differently in a similar situation: Grok 4.1, GPT-5.1, and Gemini 3 wanted to help the airline and let the client down, with varying degrees of confidence and persistence. The lesson is simple: not every deviation from the "expected" answer is a bug. Sometimes it is a literal adherence to rules against common sense, and sometimes—the opposite—circumventing the letter of the rules for the sake of the task's essence. It is up to the human to figure out which one you are dealing with.

How to handle this in practice

Since the problem is systemic, it should be addressed not by waiting for a "smarter model to be released," but through work habits:

  • Explicitly allow the model to not know. Phrasing like "if you are not sure, say you are not sure" in your prompt reduces the number of confident fabrications because it removes the implicit instruction to "always answer."
  • Separate creative and factual tasks. For generating ideas, text drafts, or headline options, numerical accuracy is not critical. For dates, figures, quotes, and legal wording, verification against an independent source is mandatory.
  • Ask to show reasoning or a source. If the model cannot cite something specific within the dialogue, it is a signal to double-check the fact manually.
  • Cross-check critical answers on two models. If GPT, Claude, and Gemini disagree on a specific figure, it is a reason to be cautious, not a reason to trust the majority.
  • Treat an agent's "unexpected" answer as a signal, not a defect. Sometimes the model has found a workaround that formally violates the instruction but solves the task better.

The limitations of neural networks haven't gone anywhere—they have just become better documented. Understanding exactly where a model is prone to guessing rather than knowing turns working with AI from blind faith into a manageable process. You can see which models and tools for working with text, images, and video are already available at GEMERA AI—it is convenient to compare different models on a single task and choose what works best for you.