Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs

2026-04-14 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Hessian-Enhanced Token Attribution (HETA) is a new framework designed to interpret predictions from decoder-only autoregressive large language models (LLMs). Existing attribution methods, often designed for encoder-based architectures, struggle with the causal and semantic complexities of autoregressive generation due to their reliance on linear approximations. HETA addresses this by integrating three components: a semantic transition vector to capture token-to-token influence across layers, Hessian-based sensitivity scores for second-order effects, and KL divergence to quantify information loss from token masking. This unified approach aims for context-aware, causally faithful, and semantically grounded attributions. The authors also introduced a benchmark dataset for evaluating attribution quality in generative settings. Empirical evaluations show HETA outperforms current methods in faithfulness and alignment with human annotations.

Key takeaway

For research scientists developing or deploying autoregressive LLMs, HETA offers a more robust and accurate method for understanding model predictions than previous techniques. You should consider integrating HETA into your interpretability toolkit, especially when causal faithfulness and semantic grounding are critical. This framework provides a new standard for evaluating and explaining complex generative model behaviors, potentially improving trust and debugging capabilities.

Key insights

HETA offers a novel, unified framework for interpreting autoregressive LLMs by combining semantic, Hessian, and KL divergence components.

Principles

• Linear approximations are insufficient for autoregressive LLM attribution.
• Second-order effects are crucial for causal faithfulness.
• Context-awareness improves attribution quality.

Method

HETA combines a semantic transition vector, Hessian-based sensitivity scores, and KL divergence to measure information loss for context-aware, causally faithful, and semantically grounded attributions in decoder-only LLMs.

In practice

• Use HETA for interpreting decoder-only LLM predictions.
• Evaluate attribution methods using the new benchmark dataset.

Topics

• Hessian-Enhanced Token Attribution
• Autoregressive LLMs
• Token Attribution
• Model Interpretability
• Decoder-Only Models

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs — HETA improves LLM interpretability by combining semantic flow, Hessian-based sensitivity, and KL divergence for causally faithful token attribution.
• LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data — LLMs struggle with epistemic self-awareness on structured clinical data, but cross-model attribution divergence can detect blind spots and improve reliability.
• OmniTrace: A Unified Framework for Generation-Time Attribution in Omni-Modal LLMs — OmniTrace provides a unified, generation-aware framework for attributing MLLM outputs to diverse input modalities.
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations — NLAs translate LLM activations into human-readable text, revealing internal thoughts and hidden motivations for auditing.
• Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation — Mechanism-aware evaluation using symbolic rules and interpretability better distinguishes true generalization from pattern exploitation.
• Proposing Topic Models and Evaluation Frameworks for Analyzing Associations with External Outcomes: An Application to Leadership Analysis Using Large-Scale Corporate Review Data — LLM-enhanced topic modeling improves interpretability, specificity, and polarity consistency for outcome-oriented text analysis.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.