DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

2026-04-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

DiscoTrace is a new method designed to identify and represent the rhetorical strategies employed by both humans and Large Language Models (LLMs) when answering information-seeking questions. This method models answers as a sequence of discourse acts related to the question, coupled with interpretations of the original question, built upon rhetorical structure theory parses. Applying DiscoTrace to answers from nine distinct human communities revealed significant diversity in their preferred answer construction strategies. Conversely, LLMs demonstrated a lack of rhetorical diversity in their responses, even when explicitly prompted to emulate specific human community guidelines. Furthermore, LLMs consistently favored breadth, addressing question interpretations that human answerers typically chose to omit.

Key takeaway

For AI Engineers developing LLM-based question answering systems, you should recognize that current LLMs lack the rhetorical diversity of human communities and tend to over-address question interpretations. Your development efforts should focus on integrating context-aware strategies to enable LLMs to produce more pragmatically appropriate and diverse answers, rather than simply mimicking human guidelines.

Key insights

DiscoTrace reveals LLMs lack rhetorical diversity and over-address question interpretations compared to human answerers.

Principles

• Human communities exhibit diverse answering strategies.
• LLMs prioritize breadth over rhetorical nuance.

Method

DiscoTrace represents answers as sequences of question-related discourse acts and question interpretations, annotated on rhetorical structure theory parses.

In practice

• Analyze LLM output for rhetorical strategy gaps.
• Develop LLMs with context-aware answering strategies.

Topics

• DiscoTrace
• Rhetorical Strategies
• Information-Seeking QA
• Large Language Models
• Human-AI Comparison

Best for: AI Engineer, Machine Learning Engineer, Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering — LLMs lack rhetorical diversity and systematically over-address question interpretations compared to human answerers.
• GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations — LLM agent memory systems perform poorly in multi-party conversations, often outperformed by simple baselines.
• The State of Information Retrieval in 2026 — Information retrieval has evolved into a reasoning problem, integrating LLMs and agentic search for complex, context-aware query processing.
• Rhetorical Questions in LLM Representations: A Linear Probing Study — LLMs encode rhetorical questions via multiple linear directions, emphasizing distinct cues rather than a single shared representation.
• Disco-RAG: Discourse-Aware Retrieval-Augmented Generation — Explicitly modeling discourse structure and planning significantly improves RAG system performance, coherence, and factual consistency.
• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual and multicultural reasoning in grounded, real-world tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.