SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection

2026-06-11 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new diagnostic measure, SICI (Stance Inference Complexity Index), has been introduced to quantify the semantic-pragmatic burden in target-text pairs for LLM stance detection. This seven-dimensional index predicts LLM accuracy more effectively than surface-level proxies and demonstrates substantial cross-scorer reliability ($α=0.771$) across datasets like SemEval-2016 and VAST. Critically, LLM error patterns exhibit regime shifts as SICI increases: low-complexity examples often result in over-attribution, particularly for "Against" stances; intermediate examples form an unstable boundary; and high-complexity examples rapidly concentrate on "None" predictions. This phase-transition-like structure is observed across GPT-3.5, GPT-4o-mini, DeepSeek-V3, and GPT-4o, with stronger models merely shifting these complexity boundaries. A 15-method intervention study further indicates that common techniques like prompting, retrieval, and debate primarily reorient models along the attribution-abstention axis rather than resolving the fundamental high-complexity bottleneck.

Key takeaway

For NLP Engineers evaluating or deploying LLMs for stance detection, understanding the SICI framework is crucial. You should recognize that increasing semantic-pragmatic complexity causes predictable shifts in LLM error patterns, moving from over-attribution to abstention. Your intervention strategies, such as advanced prompting or retrieval, may only shift these complexity boundaries rather than fundamentally resolving high-complexity bottlenecks. Prioritize diagnosing complexity with tools like SICI to tailor your model selection and intervention efforts effectively.

Key insights

LLM stance detection errors exhibit predictable regime shifts tied to semantic-pragmatic complexity, leading to distinct error patterns.

Principles

• Semantic-pragmatic complexity predicts LLM stance detection accuracy.
• LLM errors transition from over-attribution to abstention with complexity.
• Stronger LLMs shift complexity thresholds, not resolve core issues.

Method

SICI is a seven-dimensional diagnostic measure quantifying semantic-pragmatic burden in target-text pairs to predict LLM stance detection accuracy and error regimes.

In practice

• Diagnose LLM stance detection failures using SICI.
• Identify complexity thresholds for specific LLM models.
• Recognize limits of prompting for high-complexity stance tasks.

Topics

• LLM Stance Detection
• Semantic-Pragmatic Complexity
• SICI Index
• Model Error Analysis
• Prompt Engineering
• Large Language Models

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• MENTIS: What Belief Changes Under Alignment? Measuring Multi-Scale Latent Torsion in Language Models — Alignment selectively reorganizes LLM internal geometry, with measurable torsion shifts concentrated in specific layers and concepts.
• Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications — Semantic mismatches between LLM tool descriptions and code, termed DCI, pose critical reliability and security threats.
• Month in 4 Papers (April 2026) — LLM accuracy can unexpectedly improve with "rude" prompts compared to "polite" ones.
• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — MADE offers a structured, agentic approach for fine-grained, multilingual LLM evaluation diagnosis, moving beyond simple benchmark scores.
• Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States — Linear probes in LLMs often detect task format confounds, not distinct reasoning mode representations, challenging common interpretability claims.
• Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in LLM Pipelines — Correcting LLM agent bottlenecks directly can degrade performance due to downstream modules' implicit adaptation to upstream errors.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.