Deep Dive into Semantic Formatting Score: A New Metric for Meaningful Document Formatting

2026-04-28 · Source: LlamaIndex · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, quick

Summary

Llama Index has introduced Parsbench, the first document OCR benchmark specifically designed for AI agents, which uniquely evaluates semantic formatting. Unlike other benchmarks that strip formatting as cosmetic, Parsbench recognizes that elements like strikethrough, superscript, subscript, and bold carry critical meaning for agent interpretation. The benchmark includes a "semantic formatting score" that assesses four categories: text styling (strikethrough, superscript, subscript, bold), title accuracy, LaTeX preservation, and code block detection. Each category uses both positive and negative test rules to ensure correct application and avoidance of false positives. This formatting dimension has proven to be a significant differentiator among parsers, with most scoring below 50%, while LlamaParse leads with 85.2%.

Key takeaway

For AI architects and NLP engineers building document processing agents, recognizing that formatting is semantic, not just cosmetic, is crucial. Your choice of OCR parser directly impacts an agent's ability to correctly interpret document meaning, such as distinguishing valid prices from crossed-out ones. Evaluate parsers using benchmarks like Parsbench that account for text styling, LaTeX, and code blocks to ensure your agents receive accurate, semantically rich input.

Key insights

Document OCR benchmarks must evaluate semantic formatting for accurate AI agent interpretation.

Principles

• Formatting carries semantic meaning.
• Strikethrough indicates superseded content.
• Bold highlights key values.

Method

The semantic formatting score evaluates text styling, title accuracy, LaTeX preservation, and code block detection using positive and negative test rules.

In practice

• Prioritize OCR tools preserving formatting.
• Validate parser output for semantic cues.

Topics

• Semantic Formatting Score
• Document OCR Benchmarks
• AI Agent Document Interpretation
• Text Styling Evaluation
• LaTeX Preservation

Best for: Research Scientist, AI Architect, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Introducing ParseBench: The First Document Parsing Benchmark for AI Agents — ParsBench is a new benchmark for document parsing, emphasizing semantic correctness for AI agents.
• OCR Processor - Mistral AI — Mistral AI's OCR API extracts structured text from diverse document types, preserving formatting and providing confidence scores.
• Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents — Generic layout models fail to extract meaningful data snapshots from complex institutional documents.
• Inside ParseBench How to Evaluate Document Parsing for AI Agents — Document parsing for AI agents requires specialized, multi-dimensional evaluation beyond traditional benchmarks.
• [R] IDP Leaderboard: Open benchmark for document AI across 16 VLMs, 9,000+ documents, 3 benchmark suites — An open benchmark evaluates 16 VLMs on document understanding, revealing tight performance margins among top models.
• Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection — AI-text detectability is non-monotonic, influenced by edit operations and revision history, not just AI content proportion.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LlamaIndex.