DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition

2025-08-07 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

DiZiNER (Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition) is a novel framework that enhances zero-shot Named Entity Recognition (NER) performance in Large Language Models (LLMs) by simulating human pilot annotation. It employs multiple heterogeneous LLMs as annotators and a supervisor LLM (GPT-5 mini) to analyze inter-model disagreements and iteratively refine task instructions. Across 18 NER benchmarks, DiZiNER achieved zero-shot SOTA results on 14 datasets, improving prior bests by an average of +8.0 F1 and narrowing the zero-shot to supervised performance gap from -32.0 to -20.9 F1 points. The framework consistently outperformed its GPT-5 mini supervisor by +5.0 F1, indicating that performance gains stem from the disagreement-guided instruction refinement rather than the supervisor's inherent capacity. Ablation studies confirmed the importance of annotator diversity and aligning refinement with the final task objective.

Key takeaway

For AI Engineers developing zero-shot NER solutions, DiZiNER offers a robust, fine-tuning-free approach to significantly improve model accuracy. You should consider implementing a disagreement-guided instruction refinement loop with diverse LLM annotators to enhance performance, especially for complex or ambiguous entity types. This method can reduce the performance gap with supervised systems without requiring extensive human-labeled data.

Key insights

Simulating human pilot annotation with LLMs to refine instructions significantly boosts zero-shot NER performance.

Principles

• Inter-model disagreement drives instruction refinement.
• Annotator diversity is crucial for effective updates.
• Agreement metrics correlate strongly with NER performance.

Method

DiZiNER iteratively refines NER instructions by having multiple LLMs cross-annotate texts, analyzing their disagreements to identify hotspots, and then using a supervisor LLM to update common and model-specific instructions.

In practice

• Use heterogeneous LLM pools for annotation tasks.
• Prioritize instruction refinement for span boundary and type disambiguation.
• Monitor inter-model agreement as a proxy for performance.

Topics

• DiZiNER Framework
• Zero-shot Named Entity Recognition
• Large Language Models
• Instruction Refinement
• Pilot Annotation Simulation

Code references

• SiunKim/diziner-ner

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition — Simulating human pilot annotation with LLMs to refine instructions significantly boosts zero-shot NER performance.
• Retrieval-Augmented Self-Taught Reasoning Model with Adaptive Chain-of-Thought for ASR Named Entity Correction — RASTAR improves ASR named entity correction by combining robust NER with adaptive, self-taught reasoning.
• Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats — Lightweight LLMs offer competitive biomedical NER performance, with specific output formats proving more effective than multi-format tuning.
• On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance — LLM annotation reliability hinges on definition alignment, not just text memorization, as prompt-based corrections are largely ineffective.
• Draft-OPD: On-Policy Distillation for Speculative Draft Models — Draft-OPD improves speculative draft models by learning from their own policy's errors during target-assisted rollouts.
• No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No-Resource Languages — Instruction transferring on pre-trained base models effectively specializes LLMs for no-resource languages without costly instruction fine-tuning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.