Scientists have identified the neurons that cause AI to confidently lie to users

Scientists have identified the neurons that cause AI to confidently lie to users

While some celebrate the latest model release, others are counting the damage caused by its errors. Last week, news emerged of a case where a chatbot hallucination nearly triggered a diplomatic conflict between the US and China. Simultaneously, researchers from Tsinghua University published a paper identifying, for the first time, a specific mechanism within model architecture responsible for false responses. Meanwhile, the database of legal cases maintained by researcher Damien Charlotin grew to thousands of recorded incidents by mid-September, where AI failed lawyers and, in some cases, judges. This is a good opportunity to take a sober look at where neural networks actually fail and what can be done about it in practice.

Why does AI invent facts if it is "just predicting the next word"?

Precisely because it is predicting. The model does not store a database of verified facts—it generates the most probable sequence of tokens, and if a statistically plausible answer does not match reality, the system has no way of knowing. A recent study from Tsinghua went beyond general explanations and found a specific mechanism: for years, the industry explained hallucinations through three causes—dirty training data, poor reinforcement learning, or errors during the text generation stage. However, the group from Tsinghua University decided to check if there was a deeper cause within the architecture itself. The result was named the H-Neurons mechanism. The study does not promise that hallucinations will disappear tomorrow, but for the first time, it provides an address for the problem rather than a general diagnosis. The model does not know it is lying; it simply wants to please you very much. In other words, lying is built into the very logic of guessing, rather than being a separate bug that can be selectively turned off.

Can this problem simply be fixed with a model update?

No, and this is not a question of the next version. OpenAI has previously acknowledged the nature of the problem directly: the company explained in detail why language models confidently provide false information—the algorithm always builds an answer in parts, predicting the next word. Errors accumulate, and even with perfectly clean data, models would still make mistakes. That is, the longer and more complex the answer, the higher the risk that the model "made up" a detail instead of admitting uncertainty. This is a systemic property, not a temporary flaw in a specific release.

In which tasks are AI errors particularly costly?

In tasks where the result is released without human verification. The legal field is a clear example: the database of legal cases contained 2,041 cases worldwide by mid-September 2026, including 1,395 in the US, and specifically noted 32 cases where AI was used by judges, 5 cases involving prosecutors, and 15 involving experts. The problem is not limited to lawyers: errors within courts themselves are particularly sensitive. In Mississippi, a federal judge's clerk used Perplexity to prepare an order, after which non-existent names and legal grounds were discovered in the text. In the Russian context, the topic is also becoming relevant—lawyers are already warning that errors in working with AI threaten real fines, not just reputational losses.

How to notice that you are facing a hallucination rather than a fact

The main signal is excessive confidence combined with verifiable details that cannot be confirmed. Practical advice: if a link does not open or does not exist, you are facing a hallucination. Use AI as a draft, not as an expert; it is good for structuring and generating ideas, but final fact-checking must be done by a human. This is especially true for links, quotes, dates, and statistics—anything that is easy to forge in form but impossible to verify without going outside the dialogue with the model.

Are there ways to reduce the risk in practice, not just theoretically?

Yes, and they are quite down-to-earth:

  • Break long prompts into short ones—the longer the model's response, the higher the probability of accumulated error.
  • Ask for sources and verify them separately, rather than trusting the formatting—a confident tone and numbers do not prove anything on their own.
  • Use the RAG approach—that is, connect the model to real documents instead of relying on its memory: without external context, the risk of errors in factual queries is higher.
  • Keep a human in the loop for final verification in any task where an error costs money, reputation, or legal consequences.

Does all this mean that neural networks cannot be trusted at all?

No, but trust must be calibrated to the task. For a text draft, structuring ideas, or brainstorming, the risk of hallucination is not critical—it is easy to catch during editing. But for a legal opinion, a medical recommendation, or a final fact in a publication, the risk is unacceptable without human verification. The difference is not in the model itself, but in how high the cost of an error is in a specific case.

If you prefer a sober approach, try different models via GEMERA AI and compare how GPT, Claude, and Gemini handle your specific tasks. It is easier than guessing in advance where a particular model will be stronger and where it will fail.