LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

Legato 2 is a novel pipeline designed for extracting symbolic notation and semantic knowledge from sheet music images. This system introduces the first large-scale neural model for optical music recognition (OMR) that processes notation sequentially on a system-by-system basis, following horizontal lines, which significantly improves scaling for arbitrarily long musical inputs. Uniquely, Legato 2 is also the first OMR model capable of generating symbolic transcriptions that incorporate embedded textual content, such as titles and annotations. The pipeline integrates system-level segmentation with an autoregressive vision-language model to effectively capture both local notation details and overall score structure. Across various datasets, Legato 2 consistently surpasses prior models, establishing new leading performance in both OMR and downstream sheet music understanding. Furthermore, its symbolic transcriptions enhance visual inputs for frontier language models, leading to improved interpretation of complex musical documents.

Key takeaway

For Machine Learning Engineers developing optical music recognition systems, Legato 2 offers a significant advancement. You should consider adopting its system-by-system sequential processing and autoregressive vision-LM architecture to improve scalability for long inputs and accurately extract embedded textual content. This approach can enhance your models' ability to interpret dense musical documents and achieve new performance benchmarks in OMR and downstream understanding tasks.

Key insights

Legato 2 is a novel OMR pipeline that sequentially processes sheet music, extracts embedded text, and achieves new leading performance.

Principles

• Sequential OMR processing scales better.
• Vision-LMs capture notation and score structure.
• Symbolic OMR output aids language models.

Method

Legato 2 employs a pipeline combining system-level segmentation with an autoregressive vision-language model. This approach processes sheet music sequentially, system-by-system, to extract both local notation details and overall score structure, including embedded text.

In practice

• Transcribe long musical scores.
• Extract embedded text from music.
• Improve LM music interpretation.

Topics

• Optical Music Recognition
• Multimodal AI
• Vision-Language Models
• Sheet Music Analysis
• Neural Networks
• Sequential Processing

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding — LEGATO 2's sequential OMR and multimodal transcription set new benchmarks for sheet music recognition and understanding.
• A Dataset for the Recognition of Historical and Handwritten Music Scores in Western Notation — MusiCorpus provides the largest dataset of historical handwritten music for Optical Music Recognition.
• Optical Music Recognition for Real-World Manuscripts with Synthetic Data — Synthetic data, combined with minimal in-domain real data, significantly improves Optical Music Recognition for diverse real-world musical manuscripts.
• Lyria 3 Pro: Create longer tracks in more Google products — Lyria 3 Pro extends AI music generation to 3-minute tracks with structural control across Google's ecosystem.
• From Textural Counterpoint to Feature Encoding: A Multi-Dimensional Machine Representation Study of Haydn's "The Lark" Integrating Electroacoustic Analysis — Integrating classical analysis, electroacoustic measurement, and machine representation enables "role perception" in music AI.
• Exploring LLMs for South Asian Music Understanding and Generation — LLMs demonstrate strong South Asian music understanding but struggle with stylistic faithfulness in generation, revealing a gap in culturally grounded modeling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.