What Aggregate Scores Hide: Per-Rule Evaluation of Russian Grammatical Error Correction

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Russian grammar correction models can improve on aggregate benchmarks while simultaneously degrading performance on specific grammar rules. This phenomenon is demonstrated through a per-rule evaluation on a diagnostic benchmark comprising 48 prescriptive rules. For instance, finetuning on synthetic data improved overall F0.5 but reduced subordinate-clause comma accuracy from 14% to 1%. This degradation remains hidden by corpus-level metrics and existing coarse tagsets, becoming visible only through rule-granularity diagnosis. To facilitate this, researchers developed a 98-category error taxonomy based on Rozental's reference grammar and SyntErr, an open-source synthetic data generator. Finetuning eight open models (0.8B–12B) using 39K synthetic examples achieved up to 75.3 F0.5, approaching frontier API models while remaining suitable for on-device deployment. The taxonomy, generator, per-rule evaluation data, and training artifacts are openly released.

Key takeaway

For NLP Engineers developing or deploying Russian GEC models, relying solely on aggregate F0.5 scores is insufficient. You must implement per-rule diagnostic evaluation to uncover hidden performance regressions. For example, subordinate-clause comma accuracy dropped from 14% to 1% despite overall gains. Utilize the released taxonomy and SyntErr generator to create targeted benchmarks and ensure robust model improvement.

Key insights

Aggregate GEC scores can mask significant per-rule performance degradation, necessitating granular diagnostic evaluation.

Principles

• Aggregate metrics can obscure specific performance regressions.
• Granular, rule-based evaluation is crucial for diagnosing GEC model weaknesses.

Method

Develop a 98-category error taxonomy and use a synthetic data generator (SyntErr) with explicit per-rule distribution to create diagnostic benchmarks for fine-tuning and evaluation.

In practice

• Use SyntErr to generate synthetic data for specific grammar rules.
• Apply the 98-category error taxonomy for detailed error analysis.

Topics

• Grammatical Error Correction
• Russian Language Processing
• Diagnostic Evaluation
• Synthetic Data Generation
• Error Taxonomy
• Natural Language Processing

Best for: Research Scientist, AI Engineer, Machine Learning Engineer, AI Scientist, NLP Engineer

Related on AIssential

• Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations — LLM judge reliability varies significantly by evaluation criterion and input difficulty, not just the judge itself.
• Twenty Years of HAREM: A Reproducible Audit and Reassessment of Portuguese Named Entity Recognition — Data contamination in benchmarks significantly distorts model performance and masks true generalization capabilities.
• From Logistic Regression to GPT-2: Building a Complete Spam Detection & Sentiment Analysis Pipeline — For imbalanced text classification, prioritize F1-score and Precision-Recall AUC over accuracy and ROC-AUC.
• RoLegalGEC: Legal Domain Grammatical Error Detection and Correction Dataset for Romanian — RoLegalGEC provides the first Romanian legal grammatical error dataset and evaluates neural models for its use.
• Use of Machine Learning techniques and Large Language Models for automatic evaluation of Celpe-Bras exam texts — BERT adaptations achieved the best automatic evaluation for Celpe-Bras texts, but at a higher computational cost.
• A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision — Current multimodal anomaly detection systems exhibit superficial text guidance, often relying more on visual features than language conditioning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.