Featuring Every Eval Ever Results on Hugging Face Model Pages

2026-01-20 · Source: Hugging Face - Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

Hugging Face Community Evals and Every Eval Ever (EEE) are now intercompatible, enabling standardized reporting and interpretation of AI model evaluation results. Launched in February 2026, EEE, a project of the EvalEval Coalition, provides a unified JSON schema to capture comprehensive evaluation details, addressing issues like scattered results and inconsistent scores (e.g., LLaMA 65B MMLU scores varying from 63.7 to 48.8). The EEE datastore on Hugging Face currently hosts approximately 229,000 evaluation results across over 22,000 models and 2,200 benchmarks, collected from 31 reporting formats. This integration allows evaluators to submit EEE records to Hugging Face Community Evals via an automated converter tool. This ensures evaluation scores appear on Hugging Face model pages and benchmark leaderboards, complete with a source badge linking back to the full, detailed EEE record, enhancing trust and transparency for users, researchers, and policymakers.

Key takeaway

For AI scientists and ML engineers reporting model benchmarks, you should adopt the Every Eval Ever (EEE) JSON schema for your evaluation results. This ensures your scores are consistently structured and verifiable. Utilize the provided converter tool to seamlessly publish these results to Hugging Face model pages and leaderboards. This links back to comprehensive EEE records. It enhances the transparency and trustworthiness of your reported model capabilities, crucial for broader community adoption and policy discussions.

Key insights

Standardized evaluation reporting via EEE and Hugging Face integration improves AI model transparency and comparability.

Principles

• Standardize evaluation metadata.
• Link raw results to model pages.
• Verify evaluation sources.

Method

Submit full evaluation records to the EEE datastore. Use the community_evals_converter.py tool to process collections, generate local YAML previews, and open pull requests to Hugging Face Community Evals after review.

In practice

• Use EEE schema for evaluations.
• Automate Hugging Face PRs with converter.
• Check for score conflicts before submission.

Topics

• AI Model Evaluation
• Hugging Face Community Evals
• Every Eval Ever
• Evaluation Standardization
• Benchmark Reporting
• Model Transparency

Code references

• evaleval/every_eval_ever

Best for: AI Scientist, Machine Learning Engineer, Policy Maker

Related on AIssential

• Community Evals: Because we're done trusting black-box leaderboards over the community — Decentralized, transparent evaluation reporting on Hugging Face Hub aims to bridge the gap between benchmarks and real-world AI performance.
• Structured outputs — Ollama now supports constraining LLM outputs to a JSON schema for reliable, consistent structured data.
• olmo-eval: An evaluation workbench for the model development loop — olmo-eval streamlines LLM development by integrating continuous, flexible, and reproducible evaluation into the training loop.
• Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models — FEPBench offers a fine-grained, atom-set-based benchmark for T2I models generating scientific illustrations, revealing current limitations in text, reasoning, and precision.
• The Anatomy of an LLM Benchmark — Effective LLM benchmarking requires continuous evolution, rigorous data quality, and dynamic evaluation methods to counter rapid model saturation.
• Automating LLM Post-Training with Hugging Face’s ml-intern — Hugging Face's ml-intern models ML research as a multi-step workflow, not just a tool-calling chat assistant.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hugging Face - Blog.