LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study investigates the effectiveness of off-the-shelf large language models (LLMs) for automating text annotation in specialized legal domains and underrepresented languages, specifically Danish asylum decisions. Researchers introduce RAB-Cred, a Danish text classification dataset with expert annotations, annotator confidence, and asylum case outcomes. The work benchmarks 21 open-weight models and 30 system-user prompt combinations for zero-shot and few-shot classification, focusing on identifying the presence and sentiment of credibility assessments. Evaluation extends beyond aggregated metrics to analyze error consistency across LLMs, inter-class confusion, correlation with human confidence, and the severity of mistakes. Results indicate LLMs offer potential for cost-effective labeling but are imperfect and inconsistent, necessitating evaluation beyond a single model's predictions. The RAB-Cred dataset and code are publicly available.

Key takeaway

For legal NLP engineers or researchers automating text annotation in specialized domains like asylum decisions, relying solely on a single LLM or basic performance metrics is insufficient. You should systematically benchmark multiple LLM and prompt combinations, conducting detailed error analysis to understand consistency, inter-class confusion, and the severity of mistakes. This approach ensures more robust and reliable annotation pipelines, especially for critical applications involving underrepresented languages.

Key insights

LLMs can annotate specialized legal texts but require careful, multi-faceted evaluation beyond simple metrics due to inherent inconsistency.

Principles

• LLM effectiveness varies by domain and language.
• Error analysis is crucial beyond aggregate metrics.
• Single LLM predictions can be inconsistent.

Method

Benchmark 21 open-weight LLMs and 30 prompt combinations on a novel legal NLP task using zero-shot and few-shot classification, then analyze errors beyond aggregated metrics.

In practice

• Use RAB-Cred for Danish legal NLP tasks.
• Evaluate LLM annotators beyond accuracy.
• Compare multiple LLM/prompt combinations.

Topics

• Large Language Models
• Text Annotation
• Legal NLP
• Danish Language
• Asylum Decisions
• Credibility Assessment
• Error Analysis

Code references

• glhr/RAB-Cred

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling — NL-to-FOL benchmarks contain significant errors, distorting model evaluation, but LLM-assisted review can efficiently correct them.
• Auditing the Evaluators: How Far Can Automatic Evaluation Go in Assessing Portuguese Financial Texts? — Automatic and LLM-as-a-judge metrics struggle with Portuguese financial texts, requiring domain-specific evaluation.
• Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation — A new framework evaluates LLM summaries of parliamentary debates by preserving argumentative reasoning.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.
• A Larger Annotated Corpus of Portuguese Coreference — LLMs can annotate Portuguese coreference, but a new LLM-based system did not outperform existing rule-based methods.
• Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale — Human subjectivity annotation is inherently variable and influenced by context, while LLMs offer plausible but distinct scoring.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.