MortarBench: Evaluating Mortgage Loan Origination Agents

2026-06-17 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, FinTech & Digital Financial Services · Depth: Expert, quick

Summary

MortarBench is a new public benchmark designed to evaluate mortgage loan origination agents, addressing a critical gap in assessing AI systems used by lenders. This benchmark employs a financial data synthesis and mutation pipeline to generate diverse examples, ensuring broad edge case coverage and alignment with real-world distributions and questions. Initial evaluations reveal that state-of-the-art large language models perform poorly, with closed-source models achieving a maximum of 77.1% exact match accuracy. Furthermore, MortarBench uncovered systematic biases in LLM perception of "foreignness" linked to non-English names. To mitigate these issues, the authors introduce CRIT, a confidence calibration framework, which boosts accuracy to 80.5% while simultaneously enhancing risk management steering and reducing observed biases.

Key takeaway

For AI Scientists and Machine Learning Engineers developing financial agents, you should integrate robust benchmarking like MortarBench into your evaluation pipelines. Your current LLM-based systems likely suffer from significant accuracy limitations (below 77.1%) and systematic biases, particularly concerning non-English names. Implement confidence calibration frameworks, such as CRIT, to improve accuracy to 80.5% and enhance risk management, ensuring fairer and more reliable loan origination decisions.

Key insights

MortarBench reveals LLM weaknesses and biases in mortgage loan origination, improved by CRIT's calibration.

Principles

• LLMs exhibit systematic biases with non-English names.
• Public benchmarks are crucial for AI in critical financial processes.
• Confidence calibration can improve LLM accuracy and reduce bias.

Method

MortarBench uses financial data synthesis and mutation to generate diverse, real-world aligned loan origination scenarios for agent evaluation. CRIT framework increases accuracy and reduces bias through confidence calibration.

In practice

• Use MortarBench to evaluate mortgage loan agent performance.
• Implement confidence calibration to mitigate LLM biases.
• Test LLMs for "foreignness" bias with non-English names.

Topics

• Mortgage Loan Origination
• LLM Evaluation
• AI Benchmarking
• Algorithmic Bias
• Confidence Calibration
• Financial AI Agents

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Ethicist

Related on AIssential

• Precision over Probability: Orchestrating AI Agents for High-Stakes Finance — Decoupling LLM reasoning from deterministic calculation is crucial for accuracy in high-stakes financial AI.
• ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM — Frontier models struggle with agentic enterprise IT tasks, scoring below 50% on the new ITBench-AA SRE benchmark.
• Evaluating AI Agents: Techniques to Reduce Variance and Boost Alignment for LLM Judges — Aligning LLM judges with human preferences and mitigating biases is crucial for reliable AI agent evaluation.
• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory — LLMs struggle with mathematical reasoning, particularly graduate-level proofs, with GPT-5 significantly outperforming other models.
• BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents — Financial research agent evaluation requires auditing derivation steps, not just final answers, to ensure decision relevance.
• Multiagent Protocols with Aggregated Confidence Signals — Multiagent protocols aggregate individual LLM confidences into a single, discriminative system-level confidence, recovering performance losses from standard debate.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.