Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization

2026-06-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Dynamic Contextual Orthogonalization (DCO) is an inference-time intervention method designed to mitigate hallucinations in Large Language Models (LLMs). This approach posits that hallucinations arise from orthogonal noise within the residual stream's semantic manifold, where specific attention heads introduce components divergent from the context subspace. DCO leverages the input residual stream as a dynamic context anchor, performing orthogonal decomposition on attention head outputs. It then employs a layer-wise Z-score suppression mechanism to selectively attenuate outlier orthogonal components. Evaluations on Llama-3-8B and 70B across benchmarks like XSum, NQ-Swap, and IFEval demonstrate DCO's superior contextual faithfulness compared to existing baselines. Furthermore, DCO maintains high performance on knowledge-intensive tasks such as TriviaQA and TruthfulQA, effectively addressing the common trade-off between hallucination suppression and parametric knowledge retention.

Key takeaway

For machine learning engineers deploying Large Language Models, if you are struggling with persistent hallucinations, consider implementing Dynamic Contextual Orthogonalization (DCO). This method offers a computationally efficient way to improve contextual faithfulness and mitigate the trade-off between hallucination suppression and knowledge retention. Integrating DCO could significantly enhance the reliability of your LLM applications, especially for knowledge-intensive tasks.

Key insights

Hallucinations in LLMs are modeled as orthogonal noise, which Dynamic Contextual Orthogonalization (DCO) suppresses by aligning latent representations.

Principles

• Hallucinations manifest as orthogonal noise relative to the semantic manifold of the residual stream.
• Attention heads can introduce components orthogonal to the context subspace, disrupting latent representation coherence.

Method

DCO uses the input residual stream as a dynamic context anchor to perform orthogonal decomposition on attention head outputs, then applies layer-wise Z-score suppression to outlier orthogonal components.

In practice

• Apply DCO to Llama-3-8B and 70B models to enhance contextual faithfulness.
• Utilize DCO to suppress hallucinations without sacrificing parametric knowledge retention in LLMs.

Topics

• Large Language Models
• Hallucination Mitigation
• Inference-Time Intervention
• Manifold Alignment
• Orthogonal Decomposition
• Llama-3

Code references

• Harry-Miral/DCO

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models — Linguistic tone systematically influences VLM hallucination, with varying effects on frequency and intensity across models and tasks.
• Mitigating Hallucinations in Large Language Models Via Decoder Layer Skipping — DeLask mitigates LLM hallucinations by dynamically skipping deeper decoder layers identified via gradient driftance.
• On Hallucinations in Inverse Problems: Fundamental Limits and Provable Assessment Methods — AI hallucinations in inverse problems are not just model artifacts but stem from the problem's inherent ill-posedness.
• Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps — Attention-derived metrics enable efficient, inference-time hallucination detection in SpeechLLMs without gold-standard outputs.
• Type V Hallucination: When LLMs Go Off the Rails — Type V detection uses Geometric Algebra to assess LLM reasoning coherence, catching process failures missed by claim-based checks.
• Why Your AI Assistant Confidently Lies — And Why It’s Not the Data’s Fault — LLM hallucinations stem from three architectural mechanisms, not just bad training data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.