Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

2026-07-20 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A study investigated the accuracy and quality of multi-turn conversations between developers and LLM-based agents for Non-Functional Requirement (NFR) assessment, addressing a gap in current benchmarks focused solely on functional correctness. Researchers hired 49 programmers to interact with GitHub Copilot, evaluating 148 HIPAA-derived NFRs against the iTrust codebase across requirement satisfaction, reasoning, and code localization. Findings indicate that while developers generally agree with LLM assessments, the accuracy against expert ground truth remains low. User satisfaction modeling revealed that longer system responses and more information-providing turns negatively impact satisfaction, whereas proactive interactions positively influence it. These results offer insights for designing LLM dialogue systems tailored for NFR assessment.

Key takeaway

For AI Engineers developing LLM-based dialogue systems for Non-Functional Requirement (NFR) assessment, you must prioritize expert-validated accuracy over perceived developer agreement. Design your systems to deliver concise responses and incorporate proactive interaction features, as these factors significantly improve user satisfaction. Avoid overly verbose or information-dense turns to prevent negative impacts on the user experience and ensure effective NFR evaluation.

Key insights

LLM assessments for NFRs show low expert accuracy despite developer agreement, with satisfaction tied to concise, proactive interactions.

Principles

• Developer agreement with LLM NFR assessments does not equate to expert accuracy.
• User satisfaction decreases with longer LLM responses and excessive information.
• Proactive LLM interactions enhance user satisfaction in NFR assessment.

Method

49 programmers used GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase, evaluating satisfaction, reasoning, and code localization.

In practice

• Design LLM systems for NFRs with concise, direct responses.
• Implement proactive LLM features to improve user experience.
• Prioritize expert validation for NFR assessment accuracy over user perception.

Topics

• LLM Dialogue Systems
• Non-Functional Requirements
• HIPAA Compliance
• GitHub Copilot
• Software Assessment
• User Satisfaction

Best for: Machine Learning Engineer, Research Scientist, AI Product Manager, AI Scientist, NLP Engineer, AI Engineer

Related on AIssential

• Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment — LLM-based agents struggle with NFR assessment accuracy despite user agreement, with dialogue length and proactivity impacting satisfaction.
• Make way to my most dependable Intern — Effective LLM interaction mirrors guiding a capable intern, requiring clear articulation and critical assessment.
• When large language models are reliable for judging empathic communication — LLMs reliably judge empathic communication, matching experts and outperforming crowdworkers when properly benchmarked.
• Quoting Timothy B. Lee — The effective use of LLMs requires skill development and a learning curve, akin to effective management.
• When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue — Evaluating tool-augmented dialogue requires multi-dimensional assessment beyond user satisfaction or tool-calling to catch subtle errors.
• LLM-as-an-Investigator: Evidence-First Reasoning for Robust Interactive Problem Diagnosis — An evidence-first LLM agent improves problem diagnosis by iteratively questioning users to avoid premature conclusions and user-driven sycophancy.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.