It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

The EoBench benchmark systematically evaluates how 19 linguistically diverse "Expressions of Belief" (EoBs) affect large language models' (LLMs) adherence to contextual information versus their prior knowledge. This research introduces a typology spanning four dimensions—form, evidentiality, epistemic stance, and tone—to generate approximately 66,000 controlled EoB-query pairs. Evaluating 18 LLMs, including Llama3, Qwen3, and Gemma3 models ranging from 1B to 30B parameters across base and instruction-tuned variants, revealed significant behavioral variations. Larger models and instruction-tuned models generally demonstrated less context-following. Specific EoBs like child-directed or imperative forms were more persuasive, whereas counterfactuals or weak epistemic stances were less convincing. The Llama3 family also showed lower context-following rates compared to Gemma3 and Qwen3.

Key takeaway

For prompt engineers and ML engineers designing conversational AI or knowledge-grounded systems, you should recognize that an LLM's adherence to context is highly sensitive to linguistic framing. To increase the likelihood of your model accepting contextual information, consider using imperative forms or child-directed tones. Conversely, if you need the model to prioritize its prior knowledge, avoid weak epistemic stances or counterfactuals in your prompts. This understanding is crucial for robust prompt engineering and selecting appropriate models, as larger and instruction-tuned models generally exhibit less context-following.

Key insights

LLMs' context-following behavior is significantly influenced by the linguistic framing of user beliefs, not just the content.

Principles

• Instruction-tuning reduces LLM context-following.
• Larger LLM scale correlates with reduced context-following.
• Linguistic forms like imperatives or child-directed tones are more persuasive.

Method

A typology of 19 EoB types across four linguistic dimensions (form, evidentiality, epistemic stance, tone) is used with semantic tuples and templates to generate conflicting context-query pairs. LLM responses are then evaluated for context-following rate (CFR).

In practice

• Use imperative or child-directed tones for higher context integration.
• Avoid counterfactuals or weak epistemic stances when asserting facts.

Topics

• Large Language Models
• Prompt Engineering
• Context-Following Rate
• Linguistic Framing
• Instruction Tuning
• Knowledge Conflicts

Code references

• clarakuempel/EoB

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Prompt Engineer

Related on AIssential

• Production and Perception in LLMs: A Token Probability Approach — Prompt framing alone induces a production-perception distinction in LLM token probability distributions.
• Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses — LLMs significantly struggle to infer pragmatic meaning from non-verbal communication, unlike their performance with verbal cues.
• لماذا أصبحت هندسة السياق أهم من هندسة الأوامر — Context engineering, not just prompt crafting, is key for advanced LLM interaction.
• Stop Being Nice to Your Chatbot: The Surprising Science of Prompt Politeness — Polite prompts reduce LLM accuracy and increase computational cost, while direct or emotionally-charged prompts improve performance.
• When Should Models Change Their Minds? Contextual Belief Management in Large Language Models — Large language models require explicit mechanisms to manage contextual beliefs, distinguishing evidence from noise in long-horizon interactions.
• LLMs Infer Cultural Context but Fail to Apply It When Responding — LLMs infer cultural context but often fail to apply it in responses without explicit sequential prompting, revealing cultural biases.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.