IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text

2026-04-22 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Financial Regulatory AI · Depth: Expert, quick

Summary

IndiaFinBench is introduced as the first publicly available evaluation benchmark for assessing large language model (LLM) performance on Indian financial regulatory text. This benchmark addresses a gap in existing financial NLP benchmarks, which primarily use Western financial corpora. IndiaFinBench comprises 406 expert-annotated question-answer pairs derived from 192 documents from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI). It covers four task types: regulatory interpretation (174 items), numerical reasoning (92 items), contradiction detection (62 items), and temporal reasoning (78 items). Annotation quality was validated with a model-based secondary pass (kappa=0.918) and human inter-annotator agreement (kappa=0.611; 76.7% agreement). Twelve models were evaluated under zero-shot conditions, with accuracies ranging from 70.4% (Gemma 4 E4B) to 89.7% (Gemini 2.5 Flash), all significantly outperforming a 60.0% human baseline. Numerical reasoning proved the most discriminative task.

Key takeaway

For AI Engineers and Research Scientists developing LLMs for financial applications in India, IndiaFinBench offers a critical tool to assess and improve model performance. Your models should be evaluated against this benchmark to ensure accuracy in regulatory interpretation, numerical reasoning, and contradiction detection specific to Indian financial contexts. The dataset's availability allows for targeted fine-tuning, potentially leading to more robust and compliant financial AI solutions.

Key insights

IndiaFinBench provides the first public benchmark for LLMs on Indian financial regulatory text.

Principles

• Non-Western regulatory texts require specialized NLP benchmarks.
• Numerical reasoning is a highly discriminative task for LLMs.
• Expert annotation and validation are crucial for benchmark quality.

Method

The benchmark was created by expert-annotating 406 Q&A pairs from SEBI/RBI documents across four task types, then validating annotations via model-based and human agreement checks.

In practice

• Evaluate LLMs on IndiaFinBench for Indian financial use cases.
• Focus on numerical reasoning for LLM differentiation.
• Utilize the publicly available dataset for model fine-tuning.

Topics

• IndiaFinBench
• Large Language Models
• Indian Financial Regulation
• Regulatory Interpretation
• Numerical Reasoning

Code references

• rajveerpall/IndiaFinBench

Best for: AI Engineer, Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text — IndiaFinBench is a new benchmark for evaluating LLMs on Indian financial regulatory text, addressing a critical regional gap.
• FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification — Financial statement verification by LLMs demands calibrated judgment, not just arithmetic, due to real-world data nuances.
• FinTradeBench: A Financial Reasoning Benchmark for LLMs — To address the limitations of existing financial question answering benchmarks for Large Language Models (LLMs), which primarily focus on company balance sheet data and neglect market trading signals,…
• BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting — BacktestBench provides a critical benchmark for evaluating LLMs in automated quantitative trading strategy backtesting.
• TW-LegalBench: Measuring Taiwanese Legal Understanding — LLMs demonstrate proficiency in legal qualification exams but struggle with precise statutory citation and advanced reasoning in civil-law contexts.
• FinBalance: A Multi-Document Accounting Reconciliation Benchmark — LLMs struggle with multi-document accounting reconciliation, showing low accuracy and issues with document binding and consistent aggregation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.