More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdsourced Testing Assessment

2026-06-05 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, extended

Summary

This empirical study, published in June 2026, investigates the impact of agent-generated feedback on human performance in crowdsourced software testing workflows. Building on a prior multi-agent LLM-as-a-Judge assessment backbone that evaluates reports across textuality, adequacy, and competitiveness, the research conducted a controlled four-stage human-subject study with 20 testers across three real-world applications (APP1, APP2, APP3). Results show that agent-generated feedback led to immediate improvements in revised reports (e.g., Group A: +27.65% Adequacy), better first submissions on new tasks after prior feedback exposure (Group A outperformed Group B on TC, DE, TXT by +11.44%, +5.58%, +8.51% respectively), and evidence of partial skill transfer to a later application (group averages rose +7.31% TC, +7.40% ADQ from APP1 to APP3). A post-task questionnaire from 17 participants corroborated that feedback was understandable and actionable.

Key takeaway

For Directors of AI/ML overseeing crowdsourced testing initiatives, integrating LLM-powered assessment agents as feedback providers can significantly enhance report quality and tester capabilities. You should deploy these agents upstream in your workflow to offer actionable guidance on textuality and requirement coverage, moving beyond mere post-hoc judging. This approach fosters continuous improvement, reduces manual review burden, and cultivates transferable skills among your crowdworkers, leading to higher quality submissions and more efficient defect discovery.

Key insights

Agent-generated feedback, derived from LLM-based assessment, significantly improves crowdsourced test report quality and fosters skill transfer.

Principles

• Feedback improves report textuality and adequacy.
• Prior feedback enhances new task performance.
• Skill transfer is partial but meaningful across tasks.

Method

A multi-agent LLM framework assesses reports on textuality and adequacy. Feedback converts checklist findings and requirement-tree mapping into actionable revision guidance for testers.

In practice

• Integrate LLM feedback for immediate report revisions.
• Use agent guidance to improve test case clarity.
• Apply coverage feedback to address functional gaps.

Topics

• Crowdsourced Testing
• Agent-Human Interaction
• LLM-as-a-Judge
• Large Language Models
• Software Testing
• Feedback Mechanisms

Code references

• LLMJudge/MoreThanAJudge
• features/copilot

Best for: AI Scientist, Research Scientist, Software Engineer, Director of AI/ML

Related on AIssential

• Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize — Formal evaluations are crucial for AI agents, moving beyond subjective "vibe checks" to ensure reliability and performance.
• Improving Collaborative Storytelling with a Multi-Agent Framework Based on Large Language Models — An iterative LLM Writer-Editor framework significantly improves collaborative storytelling quality, even with few refinement steps.
• BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting — Automated quantitative backtesting with LLMs requires specialized benchmarks and multi-agent systems for effective strategy evaluation.
• OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment — OpenNovelty uses an LLM-powered agentic system for verifiable, evidence-based scholarly novelty assessment.
• Agent Control Patterns — Part 2: Reflection — A Simple Way to Improve Answer Quality — Separating generation from evaluation through reflection improves AI answer quality without model retraining.
• Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents — Dynamically synthesizing task-specific skills from retrieved trajectories improves LLM agent performance without parameter updates.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.