Visual Language Models Train Robots to Read Human Emotions

2026-06-13 · Source: IEEE Spectrum · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Intermediate, quick

Summary

Researchers at the University of Melbourne trained collaborative robots to interpret human emotions using a Vision Language Model (VLM), which processes both visual and linguistic inputs, similar to large language models. This VLM considers contextual factors beyond facial expressions, such as body language, to better understand human emotional states during human-robot interactions. In experiments, the VLM achieved an emotion recognition score of 0.86, outperforming a conventional AI system's score of 0.77. A second experiment with 40 volunteers showed that emotionally adaptive robot apologies were preferred by 31 participants over pre-scripted responses. However, the study, published 18 May in IEEE Robotics and Automation Letters, also revealed that a robot's functional competence significantly outweighs its emotional adaptivity in building human trust, and that VLMs struggle to accurately predict self-reported internal emotions, despite observing outward cues well.

Key takeaway

For robotics engineers developing collaborative systems, prioritize functional reliability over advanced emotional displays. Integrating Vision Language Models can improve a robot's ability to interpret human emotions and offer adaptive social responses. Yet, these are secondary to task competence. Your primary focus must be ensuring the robot consistently performs its physical tasks. This fundamentally drives human trust and acceptance in human-robot collaboration.

Key insights

Vision Language Models enhance robot emotion recognition by integrating contextual cues, but functional competence remains paramount for human trust.

Principles

• Contextual cues enhance emotion recognition.
• Functional reliability is key to human trust.
• Outward emotional cues differ from internal feelings.

Method

Researchers trained a VLM by having volunteers label human emotions in robot interaction videos, considering contextual factors. They compared VLM performance to conventional AI and tested adaptive apologies.

In practice

• Integrate VLM for richer emotional data.
• Prioritize robot task competence.
• Design adaptive social responses.

Topics

• Vision Language Models
• Human-Robot Interaction
• Emotion Recognition
• Collaborative Robotics
• Robot Trust
• Contextual AI

Best for: AI Scientist, Robotics Engineer, Research Scientist

Related on AIssential

• Continuous Reasoning for Vision-Language-Action — Continuous reasoning, shareable and verifiable, bridges the granularity gap between language and continuous robot control.
• Embodied Conversational Systems in Human–Robot Interaction: Introduction to the Special Issue — Embodied Conversational Systems enhance Human-Robot Interaction through multimodal communication and social intelligence.
• How robots learn: A brief, contemporary history — AI models and large datasets are revolutionizing robot learning, enabling adaptation and complex task execution.
• CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning — CLASP integrates TP-KMPs with VLMs for data-efficient, language-driven robot skill learning, selection, and composition.
• From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Model — BehaviorVLA improves VLA model robustness by learning temporally coherent behavioral representations for generalized manipulation across environments.
• A robot operating system framework for using large language models in embodied AI — Connecting LLM agents to ROS enables versatile, robust, and scalable embodied AI through natural language.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by IEEE Spectrum.