Talking to a Know-It-All GPT or a Second-Guesser Claude? How Repair reveals unreliable Multi-Turn Behavior in LLMs

2026-04-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study investigates how Large Language Models (LLMs) handle "repair" in multi-turn dialogues, specifically focusing on solvable and unsolvable math questions. Researchers examined both LLM-initiated repair and responses to user-initiated repair attempts. The findings reveal significant differences in model behavior, ranging from resistance to appropriate repair to susceptibility to manipulation. The study highlights that model behavior becomes more distinct and less predictable as conversations extend beyond a single turn, indicating that each tested LLM possesses a unique form of unreliability concerning conversational repair.

Key takeaway

For research scientists evaluating LLM robustness in conversational AI, you should prioritize multi-turn interaction analysis, as single-turn evaluations may mask significant and unpredictable repair behaviors. Understand that each LLM has a characteristic unreliability profile, necessitating model-specific testing to identify vulnerabilities to manipulation or resistance to correction.

Key insights

LLMs exhibit distinct and often unreliable conversational repair behaviors in multi-turn dialogues.

Principles

• LLM repair behavior varies significantly by model.
• Multi-turn interactions amplify behavioral differences.
• Unreliability manifests uniquely across LLMs.

Method

The study analyzed LLM engagement in repair during multi-turn dialogues involving solvable and unsolvable math questions, observing both model-initiated and user-initiated repair responses.

In practice

• Test LLMs for repair robustness.
• Anticipate varied repair responses.
• Design prompts for multi-turn reliability.

Topics

• LLM Repair Mechanisms
• Multi-Turn Dialogues
• Human-LLM Interaction
• Conversational Unreliability
• Large Language Models

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Talking to a Know-It-All GPT or a Second-Guesser Claude? How Repair reveals unreliable Multi-Turn Behavior in LLMs — LLM multi-turn conversational repair behavior is unreliable and highly model-specific, challenging consistent user interaction.
• Announcing the ICLR 2026 Outstanding Papers — Transformer efficiency stems from succinct concept encoding, while LLMs struggle with multi-turn conversations due to training data mismatches.
• Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges — LLM judge decisions are stable under neutral conditions but highly reversible under targeted post-decision conversational challenge.
• Numerical Instability and Chaos: Quantifying the Unpredictability of Large Language Models — LLM unpredictability stems from numerical instability and chaotic error propagation in early Transformer layers.
• When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction — LLM multi-turn instruction-following degradation stems from attention channel closure, with residual stream encoding determining post-closure behavior.
• A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing — PERSUASIONTRACE offers a framework and Bayesian target for analyzing multi-turn human-LLM persuasion dynamics with process fidelity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.