OpenGlass: Open-Source Smart Glasses for On-Device Event-Based Gesture Recognition

2025-10-09 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

OpenGlass is an open-source smart glasses platform for rapid prototyping of novel sensors and embedded machine learning algorithms. It features a modular FPC interposer, allowing flexible camera integration, including event-based (Prophesee GENX320) and frame-based cameras. A hardware-software co-designed power management system, utilizing an nRF5340 coordinator and GAP9 RISC-V SoC, enables up to 11.8 hours of continuous on-device ML from a 200 mAh battery. As a demonstration, an egocentric hand gesture recognition pipeline achieved 83.94% cross-subject accuracy (macro F1 = 0.781) on the LynX dataset using R(2+1)D, with 33.9 ms end-to-end latency on the GAP9. All hardware designs, firmware, and models are openly released.

Key takeaway

For machine learning engineers developing wearable AI applications, OpenGlass demonstrates a robust approach to on-device intelligence. You should consider its modular hardware and co-designed power management for extending battery life and integrating diverse sensors. The R(2+1)D architecture, combined with temporal augmentation and class pruning, offers a strong baseline for event-based gesture recognition, providing high accuracy and low latency on resource-constrained edge devices.

Key insights

OpenGlass provides an open-source, modular smart glasses platform for efficient on-device ML, enabling rapid sensor and algorithm prototyping.

Principles

• Modular design enhances adaptability for sensor integration.
• Hardware-software co-design optimizes power consumption.
• Event-based vision reduces bandwidth and power for wearables.

Method

Events are accumulated into polarity-separated 64x64 histogram frames, stacked into 10-frame clips, then processed by convolution-based architectures like R(2+1)D for gesture recognition.

In practice

• Utilize FPC interposers for flexible sensor integration.
• Implement event-driven wake-up for significant power savings.
• Apply temporal augmentation and class pruning for robust gesture models.

Topics

• Smart Glasses
• Event-Based Vision
• On-Device ML
• RISC-V SoC
• Gesture Recognition
• Open Hardware

Code references

• ETH-PBL/OpenGlass

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• OpenGlass: Open-Source Smart Glasses for On-Device Event-Based Gesture Recognition — OpenGlass provides an open-source smart glasses platform for efficient on-device event-based gesture recognition.
• Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices — Sensor fusion of low-resolution ToF and IR enables efficient, private gesture recognition on resource-constrained devices.
• NeuroLip: An Event-driven Spatiotemporal Learning Framework for Cross-Scene Lip-Motion-based Visual Speaker Recognition — Event cameras and specialized processing can enable robust lip-motion-based speaker recognition across diverse scenes.
• TimeLens: On-Device Artifact Recognition with Retrieval-Augmented Question Answering for the Grand Egyptian Museum — On-device artifact recognition and RAG can deliver real-time, accurate, bilingual museum guides, overcoming data and latency challenges.
• I found the best smart glasses deals right now (like this AR/XR pair for under $200) — Smart glasses offer a compelling platform for AI integration and enhanced productivity through immersive displays.
• Spatial-Temporal Decoupled Adapter for Micro-gesture Online Recognition — Decoupling spatial and temporal adaptation with adaptive augmentation significantly improves micro-gesture recognition in imbalanced datasets.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.