Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascades

2026-04-15 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

Calibrate-Then-Delegate (CTD) is a novel model-cascade approach designed for scalable Large Language Model (LLM) safety monitoring, balancing computational cost and accuracy. Unlike existing methods that delegate based on probe uncertainty, CTD utilizes a "delegation value" (DV) probe to directly predict the benefit of escalating a case to a more expensive expert model. This lightweight DV probe operates on the same internal representations as the initial safety probe. CTD enforces budget constraints by calibrating a threshold on the DV signal using held-out data and multiple hypothesis testing, providing finite-sample guarantees on the delegation rate. Evaluated across four safety datasets, CTD consistently surpasses uncertainty-based delegation at all budget levels, prevents excessive delegation, and dynamically adjusts budget allocation based on input difficulty without needing group labels.

Key takeaway

For research scientists developing scalable LLM safety systems, Calibrate-Then-Delegate offers a superior method for managing monitoring costs. You should consider integrating a delegation value probe to make more informed, budget-guaranteed escalation decisions, moving beyond less effective uncertainty-based approaches. This can significantly improve the efficiency and accuracy of your safety pipelines.

Key insights

CTD improves LLM safety monitoring by using a delegation value probe for cost-effective, instance-level escalation.

Principles

• Delegation value predicts expert benefit.
• Calibrate thresholds for budget guarantees.

Method

CTD uses a lightweight DV probe on internal representations to predict escalation benefit. A calibrated threshold, set via multiple hypothesis testing on held-out data, enforces budget constraints for instance-level delegation decisions.

In practice

• Implement DV probes for delegation.
• Use multiple hypothesis testing for budget calibration.

Topics

• Calibrate-Then-Delegate
• LLM Safety Monitoring
• Model Cascades
• Delegation Value Probe
• Budget Guarantees

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Auditing Agent Harness Safety — HarnessAudit evaluates LLM agent safety by auditing full execution trajectories, revealing mid-trajectory violations missed by output-only benchmarks.
• Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents — The Calibrate-Then-Act framework improves LLM agents' cost-aware exploration in uncertain environments.
• Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives — Tail-shape estimation in LLM evaluation is fragile, necessitating a rigorous diagnostic protocol to prevent false positives.
• Further Notes on Our Recent Research on AI Delegation and Long-Horizon Reliability — LLMs can accumulate semantic degradation in long, delegated workflows, despite strong short-horizon performance.
• CauTion: Knowing When to Trust LLMs for Ensemble Causal Discovery — CauTion integrates LLM domain knowledge with statistical ensembles using trust calibration to improve causal discovery accuracy.
• Behavior Cue Reasoning: Monitorable Reasoning Improves Efficiency and Safety through Oversight — Behavior Cues improve LLM reasoning monitorability and controllability without performance cost, enabling efficient and safer AI systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.