Ghost Context: Measuring Cross-Context Interference in Long-Context Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, medium

Summary

The paper "Ghost Context: Measuring Cross-Context Interference in Long-Context Language Models" by Rohith Namboothiri, presented at TrustNLP 2026, introduces "Ghost Context" as a phenomenon where irrelevant context spans silently influence long-context language model outputs. This leads to "misattributed grounding," where claims are supported by the wrong input context, making it undetectable by standard faithfulness metrics. The research formalizes Ghost Context and proposes a causal mask-and-rerun attribution protocol to measure it. Evaluating three widely used models across a 272-case corpus, the study found that under realistic contextual conflict, strict Ghost Context Rate (GCR) reached 38.3% for temporal contradictions, and open-ended distractor influence occurred in 20.4% of evaluations. Crucially, the paper demonstrates that these errors are remediable, with masking the highest-attributed distractor span resolving 95.5% of errors (Fix@1) with only 2.4% collateral damage. It also introduces Contextual Invariance Rate (CIR) as a robustness metric.

Key takeaway

For Machine Learning Engineers developing or deploying long-context language models, understanding "Ghost Context" is critical. Your systems implicitly trust models to use correct context, but irrelevant spans can silently misattribute grounding. You should implement attribution protocols like the causal mask-and-rerun method to detect these errors. Prioritize evaluating models with Ghost Context Rate (GCR) and Contextual Invariance Rate (CIR) to ensure reliability, as these errors are largely correctable by masking distractor spans.

Key insights

Irrelevant context in long-context LMs causes "Ghost Context" errors, leading to misattributed grounding that is detectable and correctable.

Principles

• Contextual conflict degrades LLM reliability.
• Misattributed grounding is distinct from hallucination.
• Ghost Context errors are localizable and correctable.

Method

A causal mask-and-rerun attribution protocol measures Ghost Context by identifying and masking distractor spans. Contextual Invariance Rate (CIR) quantifies robustness to irrelevant context.

In practice

• Mask highest-attributed distractor spans.
• Evaluate models using Ghost Context Rate (GCR).
• Use CIR to measure robustness to irrelevant context.

Topics

• Long-Context Language Models
• Contextual Interference
• Misattributed Grounding
• Ghost Context
• Model Reliability
• Retrieval-Augmented Generation

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models — Linguistic tone systematically influences VLM hallucination, with varying effects on frequency and intensity across models and tasks.
• Why Adding More Context Makes LLMs Less Reliable — Excessive or unstructured context degrades LLM reliability by creating competing signals that models cannot effectively prioritize.
• IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages — AudioLLMs' true context utilization is unclear; IndicContextEval provides a 7-level framework to explicitly evaluate this across 8 Indic languages.
• Pause or Fabricate? Training Language Models for Grounded Reasoning — LLMs can achieve grounded reasoning by learning to pause and clarify when information is incomplete.
• Guarantee zero “hallucinations” for your RAG Agent!* — Guaranteeing RAG "hallucination-free" output requires strict context grounding, trading abstractive freedom for verifiable traceability.
• Masking Stale Observations Helps Search Agents -- Until It Doesn't: A Regime Map and Its Mechanism — Masking stale observations in search agents yields an inverted-U accuracy gain, dependent on retriever strength and model capacity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.