Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration

2026-06-04 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

The ALMANAC dataset, introduced to enhance LLM agent collaboration, addresses a critical gap where current agents, optimized for task completion, lack the ability to continuously align mental models with human partners. Derived from the classic Map Task, a dyadic routing exercise, ALMANAC comprises 2,987 collaboration actions. Each action is meticulously annotated with theory-informed mental models, detailing participants' self-reasoning, perceived partner intent, and perceived team goal. This novel dataset aims to guide agents toward process-level collaborative competence, moving beyond mere task completion. Researchers benchmarked six LLMs using ALMANAC, demonstrating its effectiveness in evaluating models' capacity to simulate human collaborative behaviors and accurately infer their underlying mental models, thereby fostering more effective human-agent partnerships.

Key takeaway

For Machine Learning Engineers developing collaborative LLM agents, ALMANAC offers a vital resource to move beyond task-centric optimization. You should integrate this dataset to train and evaluate your models on process-level collaborative competence, specifically focusing on aligning mental models, understanding partner intent, and shared goals. Utilizing ALMANAC will enable you to develop agents that can simulate human collaborative behaviors more accurately and infer underlying mental models, fostering more effective and nuanced human-agent partnerships in complex tasks.

Key insights

ALMANAC provides action-level mental model annotations to train LLM agents for human-like collaboration.

Principles

• Continuous mental model alignment is key for collaboration.
• Agents need process-level, not just task, competence.
• Action-level mental model data guides agent collaboration.

Method

ALMANAC was built from the Map Task, collecting 2,987 dyadic routing actions. Each action received theory-informed annotations for self-reasoning, perceived partner intent, and team goal.

In practice

• Benchmark LLMs on human next-turn behavior.
• Infer agents' underlying mental models.
• Develop process-level collaborative agents.

Topics

• LLM Agents
• Human-Agent Collaboration
• Mental Models
• ALMANAC Dataset
• Data Annotation
• Collaborative AI

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration — Almanac dataset provides action-level mental model annotations to train LLM agents for effective human-agent collaboration.
• ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation — ActTraitBench quantifies the Knowledge-Decision Gap in LLMs, revealing behavioral divergence despite consistent self-reports.
• From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives? — LLMs can statistically outperform human annotators in estimating aggregate subgroup opinions under common conditions.
• CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement — CollabBench benchmarks and trains LLMs for realistic, affective collaboration in cooperative games using diverse player profiles and a hybrid reward.
• Theory of Mind in Action: The Instruction Inference Task in Dynamic Human-Agent Collaboration — Few-shot chain-of-thought prompting significantly enhances LLM Theory of Mind for collaborative task interpretation.
• Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy — LLM-integrated BEMS can equalize user performance regardless of building energy domain knowledge.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.