Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy

2026-02-18 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Energy Efficiency & Conservation · Depth: Advanced, quick

Summary

A study investigated the impact of user domain knowledge and AI literacy on the effective use of human-AI interactive Building Energy Management Systems (BEMS) integrated with Large Language Models (LLMs). Researchers conducted a role-playing experiment with 85 human subjects interacting with OpenAI GPT-4o, which simulated an LLM-integrated BEMS. Participants were tasked with identifying the top five behavioral changes for energy reduction. The study analyzed prompt-response data and participant conclusions using a hierarchical scoring framework. Participants were categorized into four groups based on self-evaluated domain knowledge and AI literacy, and Kruskal-Wallis H tests were performed across 20 quantifiable metrics. Results indicated that most participants used concise prompts (median: 16.2 words) and relied heavily on GPT's analytical capabilities. Only one metric, appliance identification rate, showed a statistically significant difference (p=0.037), driven by AI literacy, suggesting an equalizing effect of LLMs across varying expertise levels.

Key takeaway

For research scientists developing human-AI collaboration systems, this study suggests that LLM integration can mitigate the need for deep domain expertise among users. You should prioritize designing intuitive LLM interfaces that encourage concise prompting and consider that AI literacy, rather than specific domain knowledge, may be a more critical factor for user success in certain tasks, such as appliance identification.

Key insights

LLM-integrated BEMS can equalize user performance regardless of building energy domain knowledge.

Principles

• AI literacy impacts specific task performance.
• LLMs can reduce the impact of domain knowledge.

Method

A systematic role-playing experiment with 85 subjects used GPT-4o as an LLM-integrated BEMS. Prompt-response data and conclusions were analyzed with a hierarchical scoring framework and Kruskal-Wallis H tests.

In practice

• Design LLM interfaces for concise prompts.
• Focus on AI literacy training for users.

Topics

• Human-AI Collaboration
• Large Language Models
• Building Energy Management Systems
• GPT-4o
• AI Literacy

Best for: Research Scientist, AI Researcher, AI Scientist, AI Ethicist

Related on AIssential

• Are we getting better at explaining things because of our interaction with LLMs? — Crafting clear LLM prompts may enhance human explanatory and communication abilities.
• Building AI with AI — LLMs should be used to build systems, not as systems, reinforcing code's critical role.
• Written by AI, Managed by AI: Semantic Space Control and Index Sickness Elimination Across 391 Consecutive Sessions — Overly complex symbolic systems cause LLMs to lose semantic grounding, necessitating natural language-based separation for robust long-horizon collaboration.
• Using Large Language Models and Knowledge Graphs to Improve the Interpretability of Machine Learning Models in Manufacturing — Combining LLMs and KGs can generate user-friendly, context-aware explanations for ML model predictions.
• From Hallucinations to Trust: A Human-in-the-Loop Playbook — Integrating human feedback is crucial for building trust and improving the accuracy of LLM-powered agents.
• To LLM, or Not to LLM: How Designers and Developers Navigate LLMs as Tools or Teammates — LLM adoption is a sociotechnical positioning problem, not merely a technical evaluation of capability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.