Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

EAACD, an Expert-Aware Adaptive Contrast Decoding strategy, effectively mitigates hallucinations in Mixture-of-Experts (MoE) Large Language Models. Empirical studies revealed that "knowledge injection," a phenomenon used by traditional contrastive decoding, only occurs in MoE models without shared experts like LLaMA-MoE, but not in those with shared experts such as Qwen-MoE. However, the research identified that higher layers in all MoE architectures consistently show distinct expert activation patterns between factual and non-factual outputs. EAACD utilizes this by partitioning higher-layer experts into reliability groups, amplifying hallucinations in lower-reliability experts as negative references, and adaptively calibrating predictions. The method achieved state-of-the-art performance, outperforming all baselines on four datasets, including a nearly 13% improvement on HellaSwag for Qwen-MoE.

Key takeaway

For AI Scientists and Machine Learning Engineers developing or deploying MoE LLMs, this research offers a robust method to combat factual hallucinations. Your teams should consider integrating expert-aware adaptive contrast decoding (EAACD) to enhance model reliability, especially given its state-of-the-art performance without external resources. Implementing EAACD's hallucination amplification requires careful sandboxing to prevent risks, but it significantly improves factual accuracy in QA tasks.

Key insights

EAACD mitigates MoE LLM hallucinations by using expert activation differences and amplifying low-reliability expert errors for contrastive decoding.

Principles

• "Knowledge injection" is architecture-dependent in MoE models.
• Higher MoE layers show distinct expert activation for factual vs. non-factual outputs.
• Amplifying hallucinations strengthens negative references for contrastive decoding.

Method

EAACD partitions higher-layer experts by confidence/consistency, amplifies lower-reliability expert hallucinations via attention/masking, then adaptively contrasts predictions to calibrate output.

In practice

• Use expert activation patterns for MoE hallucination detection.
• Implement adaptive penalty coefficients in contrastive decoding.
• Mask influential context tokens to amplify negative references.

Topics

• Mixture-of-Experts
• LLM Hallucination Mitigation
• Contrastive Decoding
• Expert Activation Patterns
• Adaptive Decoding
• Natural Language Processing

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Expert Masking for More Efficient Expert Offloading — Fine-tuning can degrade existing factual knowledge, but strategies like parameter restriction or self-distillation can mitigate this.
• MACD: Model-Aware Contrastive Decoding via Counterfactual Data — MACD uses model feedback to create targeted counterfactual data, improving contrastive decoding for Video-LLM hallucination.
• Hallucinations in LLMs: A Deep Technical Dive into Causes, Detection, and Mitigation — LLM hallucinations are inherent to their probabilistic nature, driven by training objectives that prioritize fluency over factual accuracy.
• uir-cis at SemEval-2026 Task 12: Mitigating Prior-Induced Hallucinations in Retrieval-Augmented Reasoning via Precision-Oriented Decoding — Precision-Oriented Decoding effectively mitigates prior-induced hallucinations in LLMs for abductive event reasoning.
• ESC: Emotional Self-Correction for Reliable Vision-Language Models — Emotional signals can effectively trigger self-correction in Vision-Language Models, improving reliability without additional training.
• HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models — Hallucinated semantics in VLMs can surprisingly improve vision-language task accuracy by broadening semantic coverage during post-hallucination reasoning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.