DECODEM: Data Extraction from Corporate Organizational Documents via Enhanced Methods

2026-03-05 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, extended

Summary

DECODEM, a new set of benchmark datasets, facilitates the automated extraction of corporate governance variables from 300 corporate charters and 150 bylaws. These benchmarks pair randomly sampled documents with high-quality human annotations, enabling evaluation of large language model (LLM) extraction pipelines. The study found that automated extraction achieves high accuracy for many governance provisions, with median performance near the upper bound. However, accuracy varies, with complex provisions like shareholder meeting rights and officer-exculpation accounting for most errors. While elaborate prompting and cascading pipelines did not consistently improve performance for frontier models such as GPT-5.4 or Claude Opus 4.7, they significantly narrowed the performance gap for efficiency-oriented models like GPT-5.4-mini. This suggests pipeline design can compensate for model capability.

Key takeaway

For research scientists and legal professionals building corporate governance datasets, this study demonstrates that large language models can automate information extraction with high accuracy, significantly reducing costs and expanding research scope. You should utilize these models for standardized provisions, but remain vigilant for interpretively complex variables like shareholder meeting rights, which still require careful validation. Consider pipeline design to optimize performance, especially when using efficiency-oriented models, to ensure data quality for empirical analysis.

Key insights

Large language models can accurately extract legally meaningful variables from complex corporate documents, though performance varies by provision complexity.

Principles

• Automated legal text-to-variable translation achieves high accuracy for many provisions.
• Pipeline design can partly substitute for model capability in legal extraction tasks.
• Extraction errors are concentrated in interpretively complex legal provisions.

Method

Evaluates LLM extraction pipelines by varying prompt design (simple/detailed), task decomposition (global/topic-specific), and document handling (full document/selected excerpts) on binary classification tasks.

In practice

• Utilize DECODEM benchmarks for evaluating legal information extraction systems.
• Prioritize simpler prompts for frontier LLMs when extracting standardized legal language.
• Implement cascading pipelines to enhance performance of efficiency-oriented LLMs.

Topics

• Corporate Governance
• Legal Information Extraction
• Large Language Models
• Benchmark Datasets
• Prompt Engineering
• Empirical Legal Research

Best for: NLP Engineer, AI Scientist, Research Scientist, Legal Professional

Related on AIssential

• DECODEM: Data Extraction from Corporate Organizational Documents via Enhanced Methods — Large language models can accurately automate the extraction of legally meaningful information from complex corporate documents.
• LLM-Based Examination of Eligibility Criteria from Securities Prospectuses at the German Central Bank — LLMs can automate complex regulatory compliance by extracting and interpreting information from noisy, multilingual financial documents with high precision.
• LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification — Domain-adapted fine-tuning significantly outperforms general LLMs for specialized legal text classification.
• Agent Orchestration - LLM for Legal Metadata Extraction: A Comparative Analysis of Efficiency and Precision — Multi-agent LLM pipelines for legal metadata extraction offer conditional accuracy gains but increase complexity and cost.
• ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction — LLMs struggle with complex, enterprise-scale PDF-to-JSON extraction, especially as schema breadth increases.
• From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text — Evaluating LLMs for legal text requires both quantitative benchmarks and qualitative error analysis to reveal true capabilities.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.