Investigating first-language bias in LLM-based automated essay scoring: A cross-prompt evaluation of an open-weight AI-model on TOEFL essays

2026-07-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

A study investigated the cross-prompt generalization and first-language (L1) scoring effects of a LoRA-adapted open-weight large language model, Gemma-3-27B-it, for automated essay scoring. Fine-tuned on 480 argumentative essays, the model was evaluated on the full TOEFL11 corpus, comprising 12,100 essays from 11 L1 backgrounds across eight unseen prompts. The model achieved an overall band agreement of 77.79% and a quadratic weighted kappa of 0.702, with 99.98% adjacent-band agreement. It demonstrated robust cross-prompt generalization, maintaining accuracy across all unseen prompts. However, a systematic L1-linked scoring offset was observed, with essays from European-language backgrounds consistently receiving higher scores than those from East-Asian-language backgrounds within every proficiency band. This marks the first large-scale L1 fairness analysis of such a system.

Key takeaway

For NLP Engineers or AI Ethicists deploying LLM-based automated essay scoring systems, you must prioritize comprehensive fairness evaluations. This study highlights that even models demonstrating strong generalization can harbor systematic first-language biases, where European-language backgrounds receive higher scores than East-Asian-language backgrounds. Ensure your evaluation protocols include diverse demographic analyses to identify and mitigate such L1-linked scoring offsets before system deployment, safeguarding equitable assessment outcomes.

Key insights

A LoRA-adapted LLM shows robust cross-prompt generalization but significant first-language bias in automated essay scoring.

Principles

• LoRA adaptation enables robust cross-prompt generalization.
• First-language background can systematically bias LLM scoring.

Method

A LoRA-adapted Gemma-3-27B-it, fine-tuned on 480 argumentative essays, was evaluated on 12,100 TOEFL essays, mapping raw scores to three proficiency bands.

In practice

• Use LoRA for efficient LLM adaptation.
• Rigorously evaluate LLM scoring for L1 bias.
• Map raw scores to proficiency bands.

Topics

• Automated Essay Scoring
• Large Language Models
• First-Language Bias
• LoRA
• Gemma-3-27B-it
• TOEFL11 Corpus
• Fairness Analysis

Best for: Research Scientist, AI Product Manager, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models — LLMs encode linearly accessible essay quality representations, identifiable through specific neurons and layer-wise shifts.
• CUET-823 at SemEval-2026 Task 9: LoRA-Based Instruction Fine-Tuning of LLMs vs. Transformer Models for Bengali Polarization Detection — LoRA-based instruction fine-tuned LLMs excel at nuanced polarization detection in Bengali social media text.
• Analysing Differences in Persuasive Language in LLM-Generated Text: Uncovering Stereotypical Gender Patterns — LLMs exhibit pervasive gender-stereotypical persuasive language biases across models and languages, reflecting societal norms.
• Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise — Iterative self-reflection and revision significantly optimize LLM essay scoring rubrics for Automated Essay Scoring.
• Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards — The RLAES framework unifies LLM-based essay scoring and feedback generation via reinforcement learning with rubric-based rewards, outperforming prior methods.
• Adaptive Generation of Bias-Eliciting Questions for LLMs — Adaptive, counterfactual question generation reveals nuanced LLM biases beyond simplistic templated benchmarks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.