SAGE: Scalable AI Governance & Evaluation

2026-02-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Advanced, extended

Summary

SAGE (Scalable AI Governance & Evaluation) is a framework that operationalizes high-quality human product judgment as a scalable evaluation signal for large-scale search systems like LinkedIn Job Search and People Search. It addresses the "governance gap" where nuanced human oversight is resource-constrained, and traditional engagement proxies are often biased. At its core, SAGE employs a bidirectional calibration loop where natural-language Policy, curated Precedent, and an LLM Surrogate Judge co-evolve, transforming subjective relevance into an executable, multi-dimensional rubric with near human-level agreement (0.77 linear weighted Cohen's kappa). To achieve industrial scale, SAGE distills the high-fidelity "Teacher Judge" (GPT-o3) into a compact 8B parameter "Student Judge" at 92x lower cost, maintaining 0.72-0.73 human agreement. Deployed at LinkedIn, SAGE guided model iteration, enabled rapid offline evaluation, and detected regressions invisible to engagement metrics, ultimately driving a 0.25% lift in LinkedIn daily active users.

Key takeaway

For MLOps Engineers or AI Architects deploying semantic search systems, relying solely on engagement metrics risks missing critical relevance failures. You should implement a framework like SAGE, using a bidirectionally calibrated LLM judge to formalize human product judgment. Distill this "Teacher Judge" into a cost-efficient "Student Judge" to enable scalable, high-throughput evaluation. This approach provides precise policy oversight and accelerates model iteration, directly impacting user growth and system health.

Key insights

SAGE operationalizes human product judgment for scalable AI evaluation through calibrated LLM surrogates and distillation.

Principles

• Bidirectional calibration refines policy and judge.
• Decompose relevance for explainable failures.
• Distill LLMs for cost-effective, wide coverage.

Method

SAGE involves a bidirectional calibration loop where Policy, Precedent, and an LLM Surrogate Judge co-evolve. This calibrated "Teacher Judge" is then distilled into a cost-efficient "Student Judge" for large-scale evaluation.

In practice

• Use a graded 5-point relevance scale (0-4).
• Decompose relevance into orthogonal attributes.
• Curate small, expert-annotated precedent sets.

Topics

• AI Governance
• LLM Evaluation
• Knowledge Distillation
• Semantic Search
• Relevance Metrics
• Bidirectional Calibration

Best for: AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer, AI Architect

Related on AIssential

• AI to Learn 2.0: A Deliverable-Oriented Governance Framework and Maturity Rubric for Opaque AI in Learning-Intensive Domains — AI to Learn 2.0 governs AI-assisted work by ensuring deliverables are auditable and reflect human capability, not just AI output.
• Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code — MemAlign significantly improves LLM judge alignment with human experts using dual-memory from natural language feedback.
• A Theoretical Framework for Adaptive Utility-Weighted Benchmarking — Benchmarking AI systems should be a dynamic, human-aligned process, not a static technical exercise.
• The Sequence Opinion #806: The Emergence of the Agent-as-a-Judge: Why Evals Need a Reasoning Engine — AI evaluation is evolving from simple LLM-as-a-judge to agent-as-a-judge with reasoning capabilities.
• SAGE: A Strategy-Aware Graph-Enhanced Generation Framework For Online Counseling — SAGE integrates clinical knowledge graphs with LLMs to enhance strategy-aware, theory-driven online mental health counseling.
• Evaluate generative AI models with an Amazon Nova rubric-based LLM judge on Amazon SageMaker AI (Part 2) — Amazon Nova's rubric-based LLM judge dynamically generates task-specific evaluation criteria for generative AI models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.