HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound

2026-05-04 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Internet of Things (IoT) & Connected Devices · Depth: Advanced, quick

Summary

The HARMES dataset is a new multi-modal wearable dataset designed for Human Activity Recognition (HAR), specifically focusing on Activities of Daily Living (ADLs). It integrates three wrist-recorded modalities: Inertial Measurement Unit (IMU) motion sensing, atmospheric environmental sensors (humidity, temperature, pressure), and audio. Collected from 20 participants performing household activities in their own homes, HARMES comprises over 80 hours of recorded data, with approximately three hours of labeled activity data per participant across 15 ADL classes. This dataset is novel for combining this specific sensor trio and is nearly six times larger than the previous largest wrist-inertial-acoustic HAR dataset. Benchmark evaluations demonstrate that modality contributions are activity-dependent and offer complementary value, especially for activities ambiguous to motion data alone.

Key takeaway

For AI engineers developing Human Activity Recognition systems, HARMES offers a significant new resource. You should consider integrating environmental and audio data alongside IMU for more robust ADL recognition, particularly for activities that are difficult to distinguish with motion data alone. Explore the dataset on Zenodo and the provided code on GitHub to enhance your model's performance and generalization capabilities.

Key insights

Multi-modal wearable data improves Human Activity Recognition, especially for ambiguous daily activities.

Principles

• Modality contributions are activity-dependent.
• Complementary sensors enhance HAR accuracy.

Method

The HARMES dataset combines wrist-recorded IMU, environmental sensors (humidity, temperature, pressure), and audio from 20 participants performing 15 ADL classes in their homes.

In practice

• Utilize HARMES for multi-modal HAR research.
• Explore activity-dependent sensor fusion strategies.

Topics

• HARMES Dataset
• Human Activity Recognition
• Multi-Modal Sensing
• Wearable Sensors
• Activities of Daily Living

Best for: AI Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Opir: Efficient Multi-Task Safety Classification for Toxicity, Jailbreaks, Hate Speech, and Harmful Content — Opir provides efficient, multi-task LLM safety classification with a small deployment footprint, outperforming larger baselines.
• Wearables produce huge amounts of health data - and doctors are struggling to keep up — Healthcare's episodic model struggles to integrate the continuous, often unvalidated, data stream from consumer wearables.
• HRDexDB: A Large-Scale Dataset of Dexterous Human and Robotic Hand Grasps — HRDexDB is a multi-modal dataset for dexterous grasping, combining human and robotic hand data.
• KAN-MLP-Mixer: A comprehensive investigation of the usage of Kolmogorov-Arnold Networks (KANs) for improving IMU-based Human Activity Recognition — Strategically combining KANs for data embedding and classification with MLPs for feature mixing significantly enhances noisy IMU-based HAR.
• Human Activity Recognition Method for Moderate Violence Detection — Skeletal keypoint analysis with Random Forest can detect moderate violence in real-time surveillance.
• A foundation model for electrodermal activity data — UME is the first open-source foundation model for EDA, outperforming baselines with 20x less computation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.