To LLM, or Not to LLM: How Designers and Developers Navigate LLMs as Tools or Teammates

2026-04-20 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Human-Computer Interaction · Depth: Expert, extended

Summary

A constructivist grounded theory study, based on interviews with 33 designers and developers across three large technology organizations, reveals how practitioners navigate the integration of Large Language Models (LLMs) into workflows. The study, presented at the 2026 CHI Conference, found that decisions about LLM use are not purely technical but hinge on how LLMs are framed: as "tools" under clear human control or as "teammates" with shared or ambiguous agency. When LLMs are seen as tools, their integration is generally acceptable within existing governance structures, with humans retaining decision authority and accountability. Conversely, framing LLMs as teammates often leads to hesitation due to unclear responsibility for outcomes, though productive collaborative configurations were also observed, particularly for exploratory tasks like ideation, provided explicit oversight structures are in place. This research reframes LLM adoption as a sociotechnical positioning problem during system design.

Key takeaway

For AI Architects and Machine Learning Engineers designing systems with LLMs, your primary focus should be on clearly defining the LLM's role within the workflow. Explicitly position LLMs as "tools" with human oversight to ensure clear accountability and organizational acceptance, especially for critical tasks. If considering a "teammate" role for collaborative reasoning, ensure robust human-in-the-loop mechanisms and clear final human judgment to mitigate risks associated with ambiguous agency.

Key insights

LLM adoption is a sociotechnical positioning problem, not merely a technical evaluation of capability.

Principles

• Human control enables LLM integration.
• Ambiguous agency causes hesitation.
• Oversight structures are critical for collaboration.

Method

A constructivist grounded theory approach involved 33 semi-structured interviews with designers and developers across three large technology organizations, with iterative coding and follow-up interviews to refine analytic categories.

In practice

• Design interfaces to reinforce human control.
• Narrow LLM task scope for clarity.
• Add human review steps for accountability.

Topics

• Large Language Models
• Human-AI Teaming
• Role Positioning
• Tool Framing
• Teammate Framing

Best for: AI Architect, Machine Learning Engineer, Research Scientist, AI Engineer, Software Engineer, AI Scientist

Related on AIssential

• Building a general-purpose accessibility agent—and what we learned in the process — An accessibility agent augments human efforts by automating issue remediation and providing guidance, but requires careful design and human oversight.
• The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows — The LLM fallacy describes misattributing AI-assisted output to one's own competence, distorting self-perception.
• Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation — LLMs struggle with integrating conflicting advice for strategic resource reallocation, showing an integration-boldness tradeoff.
• The Context Tuning Playbook — LLMs are conditional probability distributions, and prompting conditions this distribution, defining the human's irreplaceable role.
• Engineering a human-aligned LLM evaluation workflow with Prodigy and DSPy — Human-aligned LLM evaluation requires iterative workflows that integrate granular human feedback to engineer context-specific metrics.
• Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition — LLM tool use with novel APIs requires specific knowledge components, with usage examples being critical, and retrieval/tuning playing complementary roles.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.