LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

LEGATO 2 is a new pipeline designed for extracting symbolic notation and semantic knowledge from sheet music images. This system introduces the first large-scale neural optical music recognition (OMR) model that processes sheet music sequentially, system-by-system, following horizontal lines of notation. This sequential approach allows for improved scaling with arbitrarily long inputs, a significant advancement over treating pages as undifferentiated images. Furthermore, LEGATO 2 is the first OMR model capable of generating symbolic transcriptions that incorporate embedded textual content, such as titles and annotations. The pipeline integrates system-level segmentation with an autoregressive vision-language model to capture both local notation details and overall score structure. Across various datasets, LEGATO 2 consistently achieves new state-of-the-art performance in both OMR and subsequent sheet music understanding tasks. It also demonstrates that its symbolic transcriptions enhance visual inputs for advanced language models, leading to better interpretation of complex musical documents.

Key takeaway

For AI Scientists and Machine Learning Engineers developing music information retrieval systems, LEGATO 2 offers a significant advancement. You should consider its sequential, system-by-system OMR approach for improved scalability with long musical inputs. Its ability to embed textual content directly into symbolic transcriptions provides richer data for downstream tasks. This could enhance your models' interpretation of complex musical documents, especially when integrating with frontier language models.

Key insights

LEGATO 2's sequential OMR and multimodal transcription set new benchmarks for sheet music recognition and understanding.

Principles

• Sequential processing improves OMR scalability.
• Multimodal OMR captures notation and text.
• Symbolic data enhances LLM music interpretation.

Method

LEGATO 2 combines system-level segmentation with an autoregressive vision-LM. It processes notation sequentially, system-by-system, to extract symbolic notation and embedded textual content.

In practice

• Transcribe sheet music with embedded text.
• Improve LLM understanding of musical scores.
• Process long musical compositions efficiently.

Topics

• Optical Music Recognition
• Multimodal AI
• Sheet Music Understanding
• Vision-Language Models
• Symbolic Notation
• Sequential Processing

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models — Robust instrument grounding in music audio-language models requires multi-axis diagnostic evaluation beyond simple binary QA.
• From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification — Multimodal fusion of visual and synthetic text features significantly enhances animal re-identification accuracy and reliability.
• OmniSch: A Multimodal PCB Schematic Benchmark For Structured Diagram Visual Reasoning — Current LMMs struggle with fine-grained visual grounding and topological reasoning for complex PCB schematics, necessitating specialized benchmarks.
• A Dataset for the Recognition of Historical and Handwritten Music Scores in Western Notation — MusiCorpus provides the largest dataset of historical handwritten music for Optical Music Recognition.
• Pitch Spelling Jazz Lead Sheets, Solo Transcriptions, Classical Piano and Monophonic Scores — A two-stage optimization algorithm jointly estimates pitch spelling and key signature from MIDI-like input for musical notation.
• I Built the Same B2B Document Extractor Twice: Rules vs. LLM — LLMs offer superior flexibility for varied document layouts compared to rigid regex rules.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.