The Bank Said GPT-5.5 Hallucinates Less. The Benchmark Said 86%. Here’s Why They’re Both Right.

2026-04-24 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, quick

Summary

GPT-5.5 was released, leading to conflicting public statements regarding its hallucination rate. Leigh-Ann Russell, CIO of the Bank of New York, praised the model's "impressive hallucination resistance," citing it as a significant improvement for scaling the bank's 220-plus AI use cases. However, six hours later, Artificial Analysis's independent AA-Omniscience benchmark reported GPT-5.5's hallucination rate at 86 percent, the highest among tested frontier models. For comparison, Claude Opus 4.7 registered 36 percent, and Gemini 3.1 Pro Preview showed 50 percent. These divergent claims highlight a critical discrepancy in how model performance is perceived and measured.

Key takeaway

For CTOs and VPs of Engineering evaluating new large language models for enterprise deployment, you must scrutinize vendor claims against independent, third-party benchmarks. Relying solely on anecdotal praise, even from prominent customers, risks integrating models with unacceptable hallucination rates, potentially undermining critical AI initiatives. Prioritize objective performance data to inform your procurement decisions and manage deployment risks effectively.

Key insights

Enterprise perception of AI model performance can significantly diverge from independent benchmark results.

Principles

• Perception often outweighs raw metrics.
• Benchmarks reveal objective model limitations.

In practice

• Cross-reference vendor claims with independent benchmarks.
• Define internal hallucination tolerance thresholds.

Topics

• GPT-5.5
• AI Hallucination
• AI Benchmarking
• Enterprise AI Adoption
• Model Evaluation

Best for: CTO, VP of Engineering/Data, Executive, Director of AI/ML, AI Product Manager, Consultant

Related on AIssential

• GPT-5.5 tops benchmarks but still hallucinates frequently at a 20 percent higher API cost — Higher compute in LLMs does not automatically reduce hallucinations or improve reasoning against nonsense.
• GPT-5.5 Outperforms (and Hallucinates), Kimi K2.6 Leads Open LLMs, AI Strains Climate Pledges, Strategic Thinking in LLMs vs. Humans — Advanced LLMs like GPT-5.5 and Kimi K2.6 push performance boundaries but reveal diverging objective and subjective evaluation outcomes.
• GPT 5.4 "we see no wall" — GPT 5.4 demonstrates human-surpassing performance in desktop navigation and expert-level task completion, signaling a new era for AI agents.
• Building an Evaluation Harness for Production AI Agents: A 12-Metric Framework From 100+ Deployments — Robust evaluation infrastructure is critical for successful, compliant AI agent deployments in production.
• Gemini 3.1 Pro and the Downfall of Benchmarks: Welcome to the Vibe Era of AI — LLM performance is increasingly domain-specific, making overall "best model" claims misleading due to specialized post-training.
• OpenAI’s Biggest Blunder: Scaling the Math That Makes ChatGPT Bluff — More computation does not mean AI stops hallucinating.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.