Encode Errors: Representational Retrieval of In-Context Demonstrations for Multilingual Grammatical Error Correction

2026-06-13 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A novel Grammatical Error Representation (GER) method significantly enhances multilingual Grammatical Error Correction (GEC) performance in large language models (LLMs) utilizing in-context learning (ICL). Current LLM few-shot GEC is suboptimal because retrieving suitable demonstrations often prioritizes semantic similarity over actual error patterns. This research demonstrates that LLMs inherently capture grammatical error information within their internal states, from which the GER, a semantically neutral encoding, is extracted. The new GER-based retrieval approach boosts ICL performance and improves correction precision across multilingual GEC datasets. For high-resource languages, 8B-sized open-source models achieve results comparable to closed-source models like Deepseek2.5 and GPT-4o-mini. Furthermore, low-resource languages see F_0.5 scores increase by up to a factor of 1.20 over baselines, offering a more precise and resource-efficient solution.

Key takeaway

For NLP Engineers developing multilingual Grammatical Error Correction (GEC) systems, you should consider implementing Grammatical Error Representation (GER)-based retrieval. This method significantly improves in-context learning precision, allowing 8B-sized open-source models to match closed-source performance for high-resource languages and substantially boost low-resource language F_0.5 scores. Adopting GER can lead to more precise and resource-efficient GEC solutions, reducing reliance on larger, more expensive proprietary models.

Key insights

LLMs' internal states contain extractable grammatical error representations (GER) that improve in-context learning for GEC.

Principles

• Grammatical errors are inherently encoded in LLM internal states.
• Semantic neutrality is key for effective error representation retrieval.

Method

Extract Grammatical Error Representation (GER) from LLM internal states, then use this semantically neutral encoding for retrieving in-context demonstrations for GEC.

In practice

• Apply GER-based retrieval to boost multilingual GEC precision.
• Utilize 8B-sized open-source models for high-resource GEC tasks.

Topics

• Grammatical Error Correction
• In-Context Learning
• Multilingual NLP
• Error Representation
• Large Language Models
• Deepseek2.5

Best for: AI Engineer, Machine Learning Engineer, Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats — Lightweight LLMs offer competitive biomedical NER performance, with specific output formats proving more effective than multi-format tuning.
• GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models — Naively using VLM-generated captions can degrade accuracy due to anchoring effects and asymmetric error types.
• Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages — ILR-informed expert judgment reveals interpretable, domain-dependent cross-lingual variation in LLM outputs.
• Retrieval-Augmented Self-Taught Reasoning Model with Adaptive Chain-of-Thought for ASR Named Entity Correction — RASTAR improves ASR named entity correction by combining robust NER with adaptive, self-taught reasoning.
• BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model — BEiTScore offers an efficient, reference-free image captioning evaluation metric, outperforming LLMs on complex tasks.
• Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors — LLM-generated SQL queries have widespread, categorized errors, requiring targeted, efficient repair solutions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.