QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard

2026-04-21 · Source: Hugging Face - Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, long

Summary

QIMMA (قِمّة), an Arabic LLM leaderboard, was launched on April 21, 2026, to systematically validate benchmarks before evaluating models, ensuring reported scores accurately reflect genuine Arabic language capabilities. It consolidates 109 subsets from 14 source benchmarks into a unified suite of over 52,000 samples across seven domains, including cultural, STEM, legal, medical, safety, poetry, and coding tasks. QIMMA is the first Arabic leaderboard to feature code evaluation, using Arabic-adapted HumanEval+ and MBPP+. A multi-stage validation pipeline, involving two state-of-the-art LLMs (Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B) and human annotators, identified systematic quality issues in existing benchmarks, such as translation problems, absent quality validation, reproducibility gaps, and coverage fragmentation. The leaderboard results, as of April 2026, show Qwen/Qwen3.5-397B-A17B-FP8 leading with an average score of 68.06, with Arabic-specialized models often outperforming larger multilingual models on cultural and linguistic tasks.

Key takeaway

For research scientists developing or evaluating Arabic LLMs, you should critically assess the quality of your evaluation benchmarks before trusting reported scores. The QIMMA leaderboard demonstrates that systematic quality issues, even in established datasets, can significantly skew results. Consider adopting a multi-stage validation process, similar to QIMMA's, to ensure your evaluations are based on high-fidelity, culturally appropriate data, particularly for code generation and culturally sensitive tasks.

Key insights

Rigorous quality validation of benchmarks is crucial for accurate and reliable LLM evaluation, especially for Arabic.

Principles

• Benchmark quality validation must precede model evaluation.
• Native content is superior to translated benchmarks.
• Transparency in evaluation outputs enhances reproducibility.

Method

QIMMA employs a multi-stage validation pipeline: automated assessment by two diverse LLMs using a 10-point rubric, followed by human review for flagged samples by native Arabic speakers, ensuring cultural and dialectal nuance.

In practice

• Prioritize native Arabic benchmarks over translated ones.
• Implement multi-model automated assessment for data quality.
• Incorporate human review for culturally sensitive content.

Topics

• QIMMA Leaderboard
• Arabic LLM Evaluation
• Benchmark Quality Validation
• Code Generation Benchmarks
• Native Arabic Content

Code references

• tiiuae/QIMMA-leaderboard
• huggingface/lighteval
• evalplus/evalplus
• mbzuai-oryx/FannOrFlop

Best for: Research Scientist, AI Scientist, NLP Engineer, Director of AI/ML

Related on AIssential

• HalluScore: Large Language Model Hallucination Question Answering Benchmark — HalluScore is a new Arabic QA benchmark for evaluating LLM hallucination, emphasizing cultural and linguistic nuances.
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias — LLM-as-a-Judge reliability is often overstated due to reliance on flawed metrics and unaddressed biases.
• Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards — LLM leaderboard rankings are biased by benchmark design, necessitating user-defined, interactive evaluation for true utility.
• CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility — A new open LLM leaderboard for European Portuguese includes novel benchmarks for cultural alignment and safeguards.
• Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware — Code-tuned general-purpose LLMs outperform domain-specific medical LLMs for T-SQL generation on manufacturing schemas.
• The Anatomy of an LLM Benchmark — Effective LLM benchmarking requires continuous evolution, rigorous data quality, and dynamic evaluation methods to counter rapid model saturation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hugging Face - Blog.