VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval
Summary
VisionAId is an Android application designed to assist over 285 million people worldwide with visual impairment by transforming a commodity smartphone into a real-time visual assistant. Unlike existing solutions often limited by predefined categories or cloud dependency, VisionAId operates primarily offline, integrating six on-device deep learning models via ONNX Runtime. These models include metric monocular depth estimation, instance segmentation, visual and facial embeddings, face detection, and a custom banknote detector. An optional cloud large language model, Google Gemini Flash, is used solely for narrative scene description and automatic object labeling. A key feature is its few-shot pipeline for personalized object retrieval, allowing users to photograph an item and later be guided to its specific instance using augmented-reality markers, spatial audio, and distance-proportional haptics. The system provides multimodal feedback, including Romanian speech synthesis, voice commands, and vibration. Performance metrics on a Samsung Galaxy S21 Ultra show INT8 quantization reduces depth latency from ~1200 ms to ~491 ms, the banknote detector achieves an mAP@50 of 0.986, and metric depth calibration is below 1 cm of error within 3 m.
Key takeaway
For AI/ML engineers developing assistive technologies, VisionAId demonstrates a powerful blueprint for creating robust, offline-first applications. You should prioritize on-device model deployment using frameworks like ONNX Runtime to ensure reliability and low latency, especially for critical real-time tasks. Consider integrating multimodal feedback and few-shot learning for personalized user experiences. This approach significantly enhances accessibility and autonomy for users with visual impairments, reducing reliance on cloud connectivity.
Key insights
VisionAId provides robust, offline-first visual assistance for the visually impaired using on-device deep learning and personalized object retrieval.
Principles
- Prioritize on-device processing for accessibility.
- Combine multiple DL models for comprehensive assistance.
- Enable few-shot learning for personalized object recognition.
Method
The system integrates six on-device deep learning models via ONNX Runtime, with an optional cloud LLM for narrative. Personalized object retrieval uses few-shot photography and multimodal guidance.
In practice
- Implement INT8 quantization for latency reduction.
- Utilize spatial audio and haptics for guidance.
- Develop custom detectors for specific needs like banknotes.
Topics
- Assistive Technology
- Visual Impairment
- On-Device AI
- Multimodal AI
- Object Retrieval
- Deep Learning Models
- ONNX Runtime
Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.