Clustering Analysis for Error Detection in Named Entity Recognition Datasets

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A novel method has been introduced for the automatic detection of annotation errors and corrections within Named Entity Recognition (NER) datasets. This approach employs a two-stage dimension reduction technique on dense sentence embeddings. Initially, the top-n principal components are identified, followed by UMAP for non-linear dimension reduction and clustering, utilizing various distance metrics. Outlier mentions are subsequently flagged for correction based on their silhouette scores. Benchmarked against the CoNLL# dataset, the method demonstrated high efficacy, with all of the top-five outliers requiring correction, and 7 of the top-10 also needing correction. Furthermore, it identified 32 of the top-50 outlier mentions as actual corrections. This technique offers a relatively low-effort solution for identifying probable annotation errors. Related code and data are publicly available.

Key takeaway

For Machine Learning Engineers or Data Scientists focused on improving Named Entity Recognition dataset quality, this method offers a robust way to automate error detection. By implementing the two-stage dimension reduction with PCA and UMAP on sentence embeddings, you can efficiently identify likely annotation errors and corrections. This approach reduces manual review effort, allowing your team to focus on more complex data challenges. Consider integrating this technique to enhance data integrity and accelerate model development cycles.

Key insights

A two-stage dimension reduction method on sentence embeddings effectively detects annotation errors in Named Entity Recognition datasets.

Principles

• Combine PCA and UMAP for effective dimension reduction.
• Silhouette scores can flag outlier mentions for correction.

Method

Apply PCA for initial dimension reduction, then UMAP for non-linear reduction and clustering, using silhouette scores to identify outlier mentions.

In practice

• Detect annotation errors in NER datasets.
• Identify corrections using outlier analysis.

Topics

• Named Entity Recognition
• Data Quality
• Error Detection
• Dimensionality Reduction
• Sentence Embeddings
• UMAP

Code references

• bltlab/clustering-for-ner

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Finding Bad Image Data using UMAP and Prodigy — UMAP-driven dimensionality reduction combined with human-in-the-loop annotation effectively identifies bad examples in large image datasets.
• Clustering Unstructured Text with LLM Embeddings and HDBSCAN — Combining LLM embeddings with UMAP and HDBSCAN enables automatic topic discovery in unstructured text.
• Data filtering methods for training language models — Label error detection methods improve model performance, particularly on small, noisy datasets, outperforming random data removal.
• Multi hash embeddings in spaCy — The report evaluates spaCy's multi-hash embedding architecture on NER datasets, validating design choices while revealing surprises.
• Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task — Supervised models with multilingual embeddings and gradient boosted trees excel at hate speech detection.
• Principal Component Analysis (PCA): Theory, Mathematics, and Applications — PCA identifies orthogonal directions of maximum variance to reduce data dimensionality and decorrelate features.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.