Using k-Shot Prompting with Large k for the Automated Scoring of a German Written Elicited Imitation Test

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Education & Learning — Educational Technology (EdTech), Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study investigated applying Large Language Models (LLMs) with k-shot prompting to automate scoring for a German Written Elicited Imitation Test (WEIT). This test evaluates literacy-dependent procedural knowledge in German as a foreign language by having test-takers reproduce briefly presented written sentences, with responses scored on an ordinal scale differentiating error types like lexical versus grammatical. Researchers observed that increasing the value of "k" within a range of 1 to 700 significantly improved scoring accuracy. However, accuracy also varied based on the specific sample drawn and across different runs of the same prompt. The k-shot prompting approach, which relies on in-context learning without an explicit scoring rubric, surpassed a baseline where only the rubric was provided to the model. Despite these gains, the LLM's performance did not exceed results achieved by prior rule-based or BERT-based models.

Key takeaway

For NLP Engineers developing automated scoring systems for foreign language assessments, consider k-shot prompting with large "k" values for initial performance gains. While this approach improves accuracy over rubric-only baselines, you should benchmark against rule-based or BERT-based models, as LLMs may not yet offer superior performance for specific tasks like German WEIT scoring. Thoroughly test prompt stability across diverse data samples and multiple runs to ensure reliable results.

Key insights

Large k-shot prompting improves LLM accuracy for German WEIT scoring but does not surpass established models.

Principles

• Increasing "k" in k-shot prompting enhances LLM accuracy.
• In-context learning can outperform explicit rubric provision.
• LLM performance varies by sample and prompt execution.

Method

The study employed k-shot prompting with an LLM, varying "k" from 1 to 700, to score German WEIT responses, comparing it against a rubric-only baseline.

In practice

• Explore large "k" values for in-context learning tasks.
• Test prompt robustness across multiple runs and samples.

Topics

• k-shot Prompting
• Large Language Models
• Automated Scoring
• German Language Assessment
• Elicited Imitation Test
• In-context Learning

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — In-context learning for Turkish LVCs is highly prompt-sensitive, with few-shot demonstrations improving LLM calibration to match supervised baselines.
• Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments? — Inline comments can be distilled into reusable natural-language rubrics, enhancing LLM-driven writing and review support.
• Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation — Soft-prompt tuning efficiently reveals true LLM knowledge by overcoming format-following limitations in benchmarks.
• Linguistic Symptoms: Augmented Generation by Retrieval and Reasoning in LLMs under Portuguese-English Variation in Medical Contexts — Linguistic variation significantly impacts smaller LLMs in medical contexts, with larger models showing greater cross-language robustness.
• Zero-Shot Goal Recognition with Large Language Models — LLM goal recognition competence varies significantly, with some models integrating evidence effectively while others rely on world-knowledge priors.
• Prompt Repetition: Google's Surprisingly Strong Baseline for Non-Reasoning LLMs — Repeating prompts can dramatically improve LLM accuracy by providing a "clean second read" with full context.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.