Not all Jensen-Shannon Divergence Estimators are Equal

2026-06-15 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Jensen-Shannon Divergence (JSD) is widely used to measure fidelity in synthetic tabular data, but its empirical estimation from finite samples is often underspecified, creating a significant measurement problem. The empirical JSD value varies based on the estimator family, sampling protocol, calibration, dimensionality, and class balance, leading to non-comparable results across different studies. Marginal-based estimators are shown to ignore dependencies in the joint distribution, severely underestimating divergence. In contrast, classifier-based estimators capture joint structure but exhibit strong estimator dependence. The analysis reveals dependence blindness in marginal estimators, prior-shift bias under class imbalance, and estimator sensitivity in high dimensions. To address prior shift, a closed-form posterior correction for classifier-based JSD estimation is derived. The study concludes that empirical JSD values are inherently protocol-dependent, making explicit specification of the estimation procedure crucial for valid comparisons. Practical guidelines and an open-source tool are offered.

Key takeaway

For Machine Learning Engineers evaluating synthetic tabular data fidelity, recognize that empirical Jensen-Shannon Divergence (JSD) values are highly protocol-dependent and often non-comparable. You should explicitly specify your JSD estimation procedure, especially when using marginal-based methods that can severely underestimate divergence. Consider employing classifier-based estimators, applying the derived posterior correction for class imbalance, and utilize the open-source tool to ensure robust and meaningful comparisons of synthetic data quality.

Key insights

JSD estimation for synthetic tabular data is protocol-dependent, requiring explicit procedure specification for valid comparisons.

Principles

• Marginal JSD estimators ignore joint dependencies.
• Classifier-based JSD estimators show strong dependence.
• Empirical JSD values are protocol-dependent.

Method

A closed-form posterior correction is derived for classifier-based Jensen-Shannon estimation to mitigate prior-shift bias under class imbalance.

In practice

• Explicitly specify JSD estimation procedures.
• Apply posterior correction for classifier-based JSD.
• Use the provided open-source JSD tool.

Topics

• Jensen-Shannon Divergence
• Synthetic Tabular Data
• Fidelity Metrics
• Estimator Bias
• Machine Learning Evaluation

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Practical estimation of the optimal classification error with soft labels and calibration — Isotonic calibration enables consistent Bayes error estimation even with corrupted soft labels, provided their order is preserved.
• Valid Inference with Synthetic Data via Task Exchangeability — Task exchangeability provides statistical principles for valid inference using synthetic data in scientific research.
• Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives — Tail-shape estimation in LLM evaluation is fragile, necessitating a rigorous diagnostic protocol to prevent false positives.
• Cross-Fitting-Free Debiased Machine Learning with Multiway Dependence — Valid DML-GMM inference is possible without cross-fitting, even with multiway clustered data.
• Error Analysis for Deep ReLU Feedforward Density-Ratio Estimation with Bregman Divergence — BDD density-ratio estimators achieve minimax optimal error bounds and mitigate the curse of dimensionality.
• A Deep Dive into Calibration of Language Models: Platt Scaling, Isotonic Regression, Temperature Scaling — LLMs are often miscalibrated; post-hoc methods like Temperature Scaling, Platt Scaling, and Isotonic Regression can align confidence with accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.