DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy

2026-04-16 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

DPrivBench is a new benchmark designed to evaluate Large Language Models' (LLMs) ability to reason about Differential Privacy (DP) guarantees. Developed by researchers from UC San Diego and OpenAI, this benchmark features 720 instances across two categories: 588 mechanism-level instances covering foundational sensitivity-based DP mechanisms like Laplace and Gaussian, and 132 algorithm-level instances derived from advanced DP research literature. Each instance asks LLMs to determine if a given function or algorithm satisfies a stated DP guarantee under specified assumptions. The benchmark is designed to cover a broad range of DP topics, diverse difficulty levels, and resist shortcut reasoning. Initial evaluations show that while top closed-source models like GPT-5-High and Gemini-3-Pro perform well on textbook mechanisms, all models struggle significantly with advanced algorithms, highlighting substantial gaps in current LLM DP reasoning capabilities.

Key takeaway

For AI Scientists and Research Scientists developing or deploying differentially private algorithms, recognize that current LLMs can assist with textbook-level DP verification but are unreliable for complex, research-level DP analysis. You should integrate LLM-based verification with formal methods or expert human review for critical applications, especially when dealing with advanced mechanisms or subtle algorithmic changes. Consider augmenting LLM prompts with relevant theorems or using in-context learning for specific, repetitive DP verification tasks to improve accuracy.

Key insights

LLMs show promise in basic DP reasoning but struggle with advanced, algorithm-specific differential privacy analyses.

Principles

• DP reasoning requires genuine understanding, not just pattern matching.
• Theorem augmentation significantly improves LLM DP reasoning accuracy.

Method

DPrivBench evaluates LLMs by presenting algorithms and claimed privacy guarantees, requiring a binary "yes" or "no" decision on correctness, covering both foundational and advanced DP concepts.

In practice

• Integrate LLMs with curated DP knowledge bases for better performance.
• Use few-shot chain-of-thought prompting for homogeneous DP tasks.

Topics

• Differential Privacy
• LLM Reasoning
• DPrivBench Benchmark
• DP Algorithm Verification
• Sensitivity-based Mechanisms

Code references

• erchiw/DPriv-Bench
• yuxiangw/autodp
• opendp/opendp
• tensorflow/privacy

Best for: AI Scientist, Research Scientist, AI Security Engineer

Related on AIssential

• DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy — LLMs struggle with advanced differential privacy reasoning, despite handling textbook mechanisms well.
• Microsoft Research Develops Novel Approaches to Enforce Privacy in AI Models — Contextual integrity in LLMs can be enforced via inference-time checks or advanced training methods to prevent sensitive information disclosure.
• POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents — POLAR-Bench diagnoses LLM agent privacy-utility trade-offs, showing frontier models significantly outperform smaller, on-device counterparts.
• Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models — Distribution shifts critically impact practical privacy in differentially private LLM adaptations, often undermining theoretical guarantees.
• Learning-Infused Formal Reasoning: From Contract Synthesis to Artifact Reuse and Formal Semantics — Learning-Infused Formal Reasoning (LIFR) integrates AI with formal methods for scalable, trustworthy verification through synthesis and artifact reuse.
• VeruSAGE: A Study of Agent-Based Verification for Rust Systems — LLMs, when paired with specialized agent systems, can effectively generate correctness proofs for Rust system software.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.