Use of Machine Learning techniques and Large Language Models for automatic evaluation of Celpe-Bras exam texts

2026-04-12 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

A study mapped and compared methods for the automatic evaluation of texts produced for the Celpe-Bras exam, Brazil's official proficiency test in Portuguese as an Additional Language. This exam requires participants to write four texts based on multimedia prompts, leading to a high volume of texts for teachers to correct and limited accessible didactic resources for students. Researchers investigated various models, including traditional machine learning algorithms and pre-trained language models like BERT, BART, and T5. The findings indicated that adaptations of the BERT model achieved the best evaluation results, though these improvements came with a considerably higher computational cost compared to other tested models.

Key takeaway

For NLP Engineers developing automated assessment tools for language proficiency exams, consider fine-tuning BERT-based models for superior accuracy in text evaluation. However, be prepared for the increased computational resources required, and evaluate if the performance gains justify the higher operational costs for your specific deployment environment and user base.

Key insights

BERT adaptations achieved the best automatic evaluation for Celpe-Bras texts, but at a higher computational cost.

Principles

• LLMs can automate language proficiency assessment.
• Model performance often correlates with computational expense.

Method

The study mapped and compared traditional machine learning algorithms and pre-trained language models (BERT, BART, T5) for automatic text evaluation in the Celpe-Bras exam context.

In practice

• Consider BERT for high-accuracy text evaluation.
• Weigh computational cost against performance gains.

Topics

• Celpe-Bras
• Automatic Text Evaluation
• Large Language Models
• BERT
• Natural Language Processing

Best for: NLP Engineer, AI Scientist, Research Scientist

Related on AIssential

• Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts — Careful feature selection yields interpretable, generalizable models for language proficiency assessment with high accuracy.
• Auditing the Evaluators: How Far Can Automatic Evaluation Go in Assessing Portuguese Financial Texts? — Automatic and LLM-as-a-judge metrics struggle with Portuguese financial texts, requiring domain-specific evaluation.
• Evaluating Automated Scoring Models on Official ENEM Essays — LLMs pretrained on mock exams significantly improve automated scoring of official ENEM essays, even with small datasets.
• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting performance.
• KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts — Domain-specific pre-training significantly enhances BERT models for Norwegian clinical NLP tasks.
• Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models — Cross-lingual fairness and tokenization efficiency are not fundamentally at odds in multilingual LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.