Using LLMs for item creation: Validating the potential of automatically generated sentence repetition test items for language assessment

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

This study investigates using GPT-4o for automatically generating test items for the Elicited Imitation Test (EIT), a sentence repetition task used in language assessment. Researchers created a parallel test form using GPT-4o and compared its linguistic and psychometric properties against two established, human-written tests. The analysis revealed some differences in grammatical structures between LLM-generated and human-written items. However, linguistic complexity did not significantly differ across the tests. Psychometric properties showed only minor variations. These findings suggest that Large Language Models like GPT-4o hold potential for Automatic Item Generation in sentence repetition tasks, which could aid standardization and parallel test creation in Second Language Acquisition (SLA) research and testing.

Key takeaway

For research scientists developing language assessment tools, this study indicates that Large Language Models like GPT-4o can significantly streamline item creation for sentence repetition tasks. You should consider integrating LLM-based Automatic Item Generation to produce parallel test forms, potentially accelerating standardization in Second Language Acquisition research. While minor grammatical and psychometric differences exist, the overall comparability suggests a robust method for expanding test item banks efficiently.

Key insights

GPT-4o can effectively generate sentence repetition test items with comparable linguistic complexity and minor psychometric differences to human-written items.

Principles

• Automatic Item Generation (AIG) with LLMs is viable for language assessment.
• LLM-generated items can achieve comparable linguistic complexity.
• Minor psychometric differences may exist in AIG items.

Method

The study created a parallel form to two validated EIT tests using GPT-4o, then analyzed linguistic and psychometric properties of the LLM-generated items.

In practice

• Generate parallel test forms for EIT.
• Support standardization in SLA testing.
• Automate item creation for language assessment.

Topics

• Large Language Models
• GPT-4o
• Automatic Item Generation
• Language Assessment
• Elicited Imitation Test
• Second Language Acquisition
• Psychometrics

Best for: AI Scientist, Research Scientist

Related on AIssential

• Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation — Multilingual-IRT extends Item Response Theory to efficiently evaluate LLMs across languages, identifying translation errors and cultural bias.
• LLMs Made Simple: Examples, Analogies & Memory Tricks — LLMs are advanced autocomplete systems predicting tokens to generate human-like language from vast training data.
• SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity — LLMs moderately generate equations from text but struggle with semantic accuracy and reliable self-evaluation.
• Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies — Formulating factuality checking as a true/false reading comprehension task with test-taking strategies improves LLM efficiency and accuracy.
• Linguistic Symptoms: Augmented Generation by Retrieval and Reasoning in LLMs under Portuguese-English Variation in Medical Contexts — Linguistic variation significantly impacts smaller LLMs in medical contexts, with larger models showing greater cross-language robustness.
• Label-Aware Pseudo-Training Sample Generation for Text Classification — A new data augmentation method uses LLMs to subtly alter sentences, improving NLP performance with limited data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.