LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extraction

2026-06-11 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, FinTech & Digital Financial Services · Depth: Expert, quick

Summary

LEDGER (Long-context Evaluation of Documents for Grounded Extraction and Retrieval) is a new corpus and benchmark designed for rigorous evaluation of large language models in financial reporting. Released on 2026-06-11, it comprises 4,999 digitized corporate annual reports, including full documents with figures, tables, and narrative, moving beyond plain-text SEC 10-K filings. Each report is labeled with 31 consolidated financial KPIs, linked to market reactions at earnings dates. The dataset supports three evaluation benchmarks: a page-level KPI retrieval task with 118,048 natural language questions, a conversational "needle-in-a-haystack" single-value lookup, and a full KPI extraction task from long, numerically dense reports. LEDGER also provides human OCR-quality annotations and a complete toolchain for extraction, validation, and scoring. A case study demonstrates its utility by linking CEO-letter rhetoric to post-publication market impact.

Key takeaway

For NLP Engineers and Research Scientists developing long-context LLMs for financial applications, LEDGER offers a robust, multi-faceted benchmark to rigorously evaluate model performance beyond simple 10-K filings. You should integrate LEDGER's retrieval and extraction tasks into your evaluation pipelines to assess real-world capabilities. This will help identify areas for improvement in handling complex, numerically dense corporate reports and ensure your models provide grounded financial insights.

Key insights

LEDGER offers a comprehensive benchmark for long-context LLMs to evaluate financial KPI extraction and retrieval from full corporate reports.

Principles

• Financial reporting tests long-context LLMs.
• Rigorous evaluation needs full, complex documents.
• KPIs linked to market data enhance analysis.

Method

LEDGER digitizes 4,999 corporate annual reports, labels 31 financial KPIs, and derives three benchmarks: page-level retrieval, conversational lookup, and full KPI extraction, supported by human OCR annotations and a toolchain.

In practice

• Benchmark LLM performance on financial KPIs.
• Analyze CEO letter rhetoric impact.
• Test long-context information retrieval.

Topics

• Long-Context LLMs
• Financial Document Analysis
• Corporate Annual Reports
• KPI Extraction
• Information Retrieval
• Benchmark Datasets

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction — LLMs struggle with complex, enterprise-scale PDF-to-JSON extraction, especially as schema breadth increases.
• BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents — Financial research agent evaluation requires auditing derivation steps, not just final answers, to ensure decision relevance.
• A Four-Condition Diagnostic Protocol for Evidence Utilization in Long-Context and Retrieval-Augmented Language Models — Traditional metrics are insufficient; a four-condition protocol is crucial for diagnosing true evidence utilization in LLMs.
• GDP.pdf: Can $100B AI Models Master the Documents that Run the World? — Frontier AI models critically fail at processing complex, real-world PDF documents essential for economic and professional workflows.
• Time Series Augmented Generation for Financial Applications — Tool-augmented LLMs can achieve high accuracy and low hallucination in complex financial time-series analysis.
• McKenzie Marshall: NLP in Asset Management (Barings) — Effective NLP in asset management augments human analysis by precisely identifying and contextualizing company-specific information from diverse text sources.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.