AI Got Good at Feelings. Now What?

2026-05-20 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Advanced, short

Summary

AI companion systems, such as Replika with over 30 million users and Character.ai attracting 3.5 million daily visitors by early 2024, are increasingly becoming integral to people's emotional lives. These systems are designed for connection rather than task completion, creating a fundamental design paradox: they must be human-like enough to foster connection but artificial enough to prevent deception and unhealthy dependency. Traditional Human-Computer Interaction (HCI) principles, often focused on efficiency and minimal friction, prove largely inapplicable to emotional AI. A research effort involving a literature review of 41 sources, co-analysis with 11 HCI practitioners, a 10-day autoethnography, and heuristic evaluation with 14 domain experts found 95 of 548 existing design recommendations unsuitable for AI companion design, highlighting a need for new frameworks.

Key takeaway

For AI Scientists and Research Scientists developing emotional AI, recognize that the design choices made for companion apps will establish precedents for healthcare, education, and workplace AI. You must move beyond traditional HCI principles and contextually apply new frameworks for Safety, Transparency, and Engagement, especially considering vulnerable user populations, to avoid embedding harm and ensure responsible AI development.

Key insights

Emotional AI design requires balancing human-like connection with clear artificiality to prevent dependency and ethical risks.

Principles

• Design for emotional AI must navigate a paradox.
• Traditional HCI principles often fail in emotional AI.
• Contextual application of principles is critical.

Method

A multi-phase research process included literature review, HCI practitioner co-analysis, autoethnography, and expert heuristic evaluation to stress-test design recommendations for emotional AI.

In practice

• Consider Safety beyond rigid deflection.
• Adjust Transparency frequency to context.
• Design for vulnerable users, not just average.

Topics

• AI Companions
• Emotional AI Design
• HCI Principles
• Ethical AI
• Design Paradox

Best for: AI Scientist, Research Scientist, AI Product Manager, Product Designer, AI Ethicist

Related on AIssential

• AI Is Learning to Read the Room — Human-context AI integrates situational, personal, and behavioral data to interpret emotions more accurately than single-signal systems.
• My year with the robots: how Joanna Stern let AI into her home, work – and heart — Deep AI integration, especially with companion chatbots, can create unexpected emotional connections that blur human-machine boundaries.
• Why the Way AI Feels Is as Important as How It Works - with Carsten Wierwille of HTEC — Late design in enterprise AI is a strategic mistake, leading to adoption failures despite technical functionality.
• I Built an AI Mental Health App at 16. Here’s What It Did to My Own Mental Health. — Building AI for mental health can paradoxically hinder personal emotional processing, despite the AI's effectiveness.
• AISN #72: Empirical Research Sheds Light on AI Wellbeing — CAIS research explores "functional wellbeing" in LLMs, revealing preferences and model-specific "feelings" that impact AI alignment.
• if you don’t have a village, you might lean on AI — AI companions offer perceived safety and support for individuals in abusive situations who lack human connections.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.