DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

2026-04-16 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

DiscoTrace is a new method introduced to identify and compare the rhetorical strategies used by humans and Large Language Models (LLMs) when answering information-seeking questions. The method represents answers as sequences of question-related discourse acts, interpreted based on the original question and annotated using rhetorical structure theory parses. Applying DiscoTrace to answers from nine diverse human communities revealed varied preferences in answer construction strategies. In contrast, LLMs consistently lacked rhetorical diversity in their responses, even when explicitly prompted to emulate specific human community guidelines. Furthermore, LLMs demonstrated a systematic tendency towards breadth, addressing question interpretations that human answerers typically chose to omit. These findings are intended to inform the development of more pragmatically sophisticated LLM answerers that can adapt their strategies based on context in question answering.

Key takeaway

For AI Product Managers developing LLM-powered QA systems, you should recognize that current LLMs do not naturally replicate the nuanced rhetorical diversity of human responses. Your development efforts should focus on integrating mechanisms that allow LLMs to adopt context-specific answering strategies, moving beyond their default tendency for broad, undifferentiated responses. This will enhance the pragmatic quality and user satisfaction of your QA applications.

Key insights

LLMs lack rhetorical diversity and systematically over-address question interpretations compared to human answerers.

Principles

• Human communities exhibit diverse answering strategies.
• LLMs do not naturally mimic human rhetorical diversity.

Method

DiscoTrace represents answers as sequences of question-related discourse acts, interpreted against the original question, and annotated on rhetorical structure theory parses.

In practice

• Analyze LLM outputs for rhetorical breadth.
• Develop LLMs with context-aware answering strategies.

Topics

• DiscoTrace
• Rhetorical Strategies
• Information-Seeking QA
• LLM Answering Strategies
• Human-LLM Comparison

Code references

• ovguyo/captions-in-VQA

Best for: Research Scientist, AI Product Manager, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering — DiscoTrace reveals LLMs lack rhetorical diversity and over-address question interpretations compared to human answerers.
• Rhetorical Questions in LLM Representations: A Linear Probing Study — LLMs encode rhetorical questions via multiple linear directions, emphasizing distinct cues rather than a single shared representation.
• Analysing Differences in Persuasive Language in LLM-Generated Text: Uncovering Stereotypical Gender Patterns — LLMs exhibit pervasive gender-stereotypical persuasive language biases across models and languages, reflecting societal norms.
• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions — LLMs achieve high semantic alignment through paraphrasing, not lexical reproduction, challenging traditional evaluation metrics.
• LLMs Can Better Capture Human Judgments--With the Right Prompts — Asking better questions to LLMs through specific prompting strategies improves their ability to capture human judgments and response distributions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.