Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity

2026-05-22 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, extended

Summary

A study compared large language models (LLMs) and fine-tuned RoBERTa for extracting 25 inferentially complex circumstances from 191,696 National Violent Death Reporting System (NVDRS) narratives. Researchers developed a "Complexity Score" algorithm to predict whether detailed prompts with full coding guidelines or simpler name-only prompts would yield better performance. A hybrid approach, which uses this algorithm to select the prompt strategy per circumstance, achieved a macro F1 score of 0.893, closely matching the oracle's 0.897. LLMs, including GPT-5.2 (0.883 F1 with complex prompts), Gemini 2.5 Pro (0.878 F1), and Llama-3 70B (0.838 F1), consistently outperformed fine-tuned RoBERTa (0.800 F1), especially for low-prevalence circumstances with fewer than 5,000 positive training instances. The Complexity Score algorithm demonstrated 87% accuracy in predicting the optimal prompt strategy for non-tie circumstances.

Key takeaway

For NLP Engineers automating data extraction from complex, imbalanced narrative datasets like NVDRS, you should adopt a hybrid model architecture. Deploy LLMs for low-prevalence, inferentially complex circumstances where training data is scarce, as they significantly outperform fine-tuned models. Conversely, retain fine-tuned models for high-prevalence, straightforward circumstances. Implement a "Complexity Score" algorithm to dynamically select the appropriate prompt strategy, optimizing performance without empirical tuning per task. This approach maximizes accuracy and efficiency across diverse data characteristics.

Key insights

LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.

Principles

• LLMs excel at semantic inference with scarce data.
• Fine-tuned models struggle with complex, rare cases.
• Prompt complexity should match inference requirements.

Method

A framework extracts definitions, guidance, and examples from coding manuals. A "Complexity Score" algorithm, based on negative example linguistic features, predicts optimal prompt strategy (simple vs. complex) for each circumstance.

In practice

• Deploy LLMs for rare, inferentially complex data.
• Use fine-tuned models for common, high-prevalence data.
• Implement Complexity Score for dynamic prompt selection.

Topics

• Large Language Models
• Natural Language Processing
• NVDRS Data Extraction
• Prompt Engineering
• Hybrid AI Architectures
• Semantic Inference

Best for: AI Engineer, Machine Learning Engineer, AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• FineREX: Fine-Tuned NER-RE for Human Smuggling Knowledge Graphs — Domain-specific fine-tuning of LLMs significantly enhances knowledge graph construction quality and efficiency for complex, jargon-heavy legal texts.
• Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries — LLM-based clinical data extraction sensitivity varies significantly with model and schema, especially regarding "absence-versus-silence" distinctions.
• Revisiting the Systematicity in Negation in the Era of In-Context Learning — LLMs partially grasp negation via in-context learning, but struggle with robust scope recognition and function vector composition for it.
• Improving Answer Extraction in Context-based Question Answering Systems Using LLMs — Fine-tuning pre-trained LLMs on benchmark QA datasets significantly improves answer extraction accuracy and contextual understanding in QA systems.
• FineREX: Fine-Tuned NER-RE for Human Smuggling Knowledge Graphs — Domain-specific fine-tuning of LLMs substantially improves knowledge graph quality and efficiency for complex legal text.
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data — Structured pedagogical formats and diverse outputs are key to high-quality synthetic pretraining data, not larger generator models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.