Shownotes

Large Language Models (LLMs) sometimes produce confident but wrong answers—what we call hallucinations. This post explores a recent OpenAI paper that explains why this happens, why it’s not actually a flaw in the models themselves, and what we can do to reduce it.

Key Points Covered

• The Problem of Hallucinations
  • LLMs often produce plausible but incorrect responses.
  • Users criticize this as a core weakness of AI systems.
• The Student Test Analogy
  • Like students on multiple-choice exams, LLMs are trained to guess when uncertain.
  • Test-taking strategy: eliminate wrong answers, then guess among the rest.
  • Guessing improves accuracy on average since there’s no penalty for being wrong.
• How Training Causes Hallucinations
  • Pre-training: LLMs learn patterns in language, not always “truth.”
  • Post-training with Reinforcement Learning from Human Feedback (RLHF): models are rewarded for correct answers, not for admitting uncertainty.
  • Saying “I don’t know” is treated the same as being wrong (a zero), which discourages caution.
• Confidence in Models
  • If you sample many outputs, models show variation: high agreement = high confidence, wide variation = uncertainty.
  • But models aren’t rewarded for expressing that uncertainty.
• The Paper’s Findings
  • Hallucinations arise naturally from statistical pressures in training.
  • Current benchmarks (MMLU, GPQA, SWE-bench, etc.) don’t reward “I don’t know” responses.
  • Only one benchmark (WildBench) partially accounts for uncertainty.
• Proposed Solutions
  • Adjust benchmarks and training to give partial credit for “I don’t know.”
  • Penalize confidently wrong answers instead of treating them the same as uncertainty.
  • Encourage models to mimic human behavior outside of tests, where showing uncertainty earns trust.
• Takeaway
  • Hallucinations aren’t a bug—they’re a byproduct of how we train and evaluate models.
  • With small shifts in incentives, future LLMs could become more reliable by admitting uncertainty.

Quotes & Highlights

• “Language models are optimized to be good test takers—and guessing when uncertain improves test performance.”
• “There’s no incentive for an LLM to say ‘I don’t know.’”
• “We don’t call it hallucination when students guess—we call it smart test-taking strategy.”
• “The shame of confidently saying something stupid is exactly what these models are missing.”

Resources Mentioned

• OpenAI research paper on LLM hallucinations
• OpenAI Blog Post
• Related methods: Intuitor and reinforcement learning from internal feedback

Summary

This post breaks down why AI language models sometimes “hallucinate”—giving confident but wrong answers. OpenAI researchers argue hallucinations aren’t a mysterious flaw, but a predictable side-effect of how LLMs are trained and tested.

Like students on multiple-choice exams, LLMs are rewarded for correct answers but never penalized for guessing. Saying “I don’t know” gets them nothing, so they guess instead—sometimes confidently wrong. This behavior boosts benchmark performance but creates trust issues in real-world use.

The solution? Change incentives. Benchmarks and training methods should give credit for expressing uncertainty and penalize confidently wrong answers. Just as humans learn the value of saying “I don’t know” in professional settings, AI systems could too—if we train them that way.

The paper suggests that with these changes, hallucinations could be significantly reduced, leading to more trustworthy AI systems.