Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code

2026-06-26 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A Wizard-of-Oz experiment with 32 software engineers investigated how the explicit labeling of code as LLM-generated influences code review behavior. Researchers used eye-tracking and exit interviews, analyzing fixation durations and saccade lengths on human-written code presented as either LLM-generated or unlabelled. The study found that while review thoroughness, measured by saccade lengths, remained unchanged, participants spent significantly more time fixating on LLM-labeled code, with increases up to 60% for complex segments. Qualitatively, 37.5% of reviewers adjusted their evaluation criteria based on their trust in AI, prioritizing logical correctness or code quality. Additionally, 14 participants utilized the embedded prompt as either a requirement specification for validation or as contextual documentation, highlighting its role as a valuable software artifact.

Key takeaway

For software engineering teams adopting LLMs for code generation, recognize that simply labeling code as AI-generated increases review time, especially for complex sections, without necessarily improving defect detection. You should implement clear, structured AI policies that specify "how" to verify LLM-generated code, including constraints on code size and submission timing. Additionally, ensure prompts are available on-demand within the development environment as metadata, enabling reviewers to validate code against original intent without disrupting workflow.

Key insights

Explicitly labeling code as LLM-generated increases review time without improving thoroughness, driven by perceived origin.

Principles

• Perceived code provenance impacts attention.
• Self-reported trust doesn't predict review thoroughness.
• AI policies need concrete verification steps.

Method

A Wizard-of-Oz experiment used eye-tracking and interviews with 32 software engineers reviewing human-written code, some segments explicitly labeled as LLM-generated, to isolate the effect of perceived origin.

In practice

• Integrate prompt-to-code traceability in IDEs.
• Define specific LLM code review guidelines.
• Monitor review time for LLM-generated code.

Topics

• Large Language Models
• Code Review
• Eye Tracking
• Software Engineering Policy
• Prompt Engineering
• Human-AI Interaction

Code references

• psf/requests
• -assistant/core

Best for: Machine Learning Engineer, NLP Engineer, AI Scientist, Software Engineer, AI Engineer, Research Scientist

Related on AIssential

• The TechBeat: Can LLMs Generate Quality Code? A 40,000-Line Experiment (1/6/2026) — LLMs generate sloppy code over time, requiring formal analysis and multi-model reviews for quality assurance.
• Multi-Model Code Review: How Developers Can Catch Better Bugs Without Drowning in AI Noise — Structured multi-model code review, using narrow lenses and consensus aggregation, enhances human judgment by reducing AI noise.
• Bigger Isn't Always Better: A Comparative Evaluation of LLMs for Automated Code Review — Smaller, cheaper LLMs can outperform larger models for automated code review, particularly on real-world data.
• Towards the Readability of LLM-Generated Codes through Multitask Representation Engineering — Multitask Representation Engineering improves LLM-generated code readability by enabling targeted control across multiple tasks.
• Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment — An agentic LLM pipeline scalably labels architectural quality, significantly improving code LLM reasoning and patch conformance.
• SED News: OpenCode, AI Code vs. Shipped Code, and the LiteLLM Breach — AI accelerates code generation, but human verification and the full software development lifecycle remain critical bottlenecks for production.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.