From BERT to T5: A Study of Named Entity Recognition

2026-05-18 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

This report details the implementation and comparison of two pretrained models, BERT and T5, for Named Entity Recognition (NER) tasks. The study fine-tuned an encoder-only BERT model using a simple classification head and weighted cross-entropy loss, while a sequence-to-sequence T5 model was fine-tuned with few-shot prompts and two distinct validation strategies. Experiments were conducted using both a 7-class and a simplified 3-class tag scheme. The research also included an ablation study to assess the impact of various hyperparameters and analyzed common error patterns observed in BERT, providing insights into the performance of both architectures for sequence labeling.

Key takeaway

For AI Engineers evaluating pretrained models for Named Entity Recognition, this study highlights that BERT, as an encoder-only model, can be effectively used with a classification head, while T5, a sequence-to-sequence model, benefits from few-shot prompting. Your choice should consider the specific task requirements and the complexity of the tag scheme, as both architectures offer distinct advantages. Review the error analysis to anticipate potential challenges with BERT.

Key insights

BERT and T5 models were fine-tuned and compared for Named Entity Recognition using different architectural approaches.

Principles

• Encoder-only models use classification heads.
• Sequence-to-sequence models benefit from few-shot prompts.

Method

The study fine-tuned BERT with weighted cross-entropy and T5 with few-shot prompts and two validation strategies, conducting an ablation study on hyperparameters.

In practice

• Consider BERT for classification-based NER.
• Explore T5 with few-shot prompting for sequence-to-sequence NER.

Topics

• Named Entity Recognition
• BERT Model
• T5 Model
• Finetuning
• Sequence Labeling

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• BERT in NLP: How an Encoder-Only Transformer Changed Language Understanding — BERT's encoder-only, bidirectional pretraining revolutionized NLP by enabling broad language understanding and easy task adaptation.
• Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats — Lightweight LLMs offer competitive biomedical NER performance, with specific output formats proving more effective than multi-format tuning.
• How many Labelled Examples do you need for a BERT-sized Model to Beat GPT-4 on Predictive Tasks? — BERT-sized models often beat GPT-4 on predictive NLP tasks with sufficient labeled data.
• Which tokens does a hybrid model predict better? — Hybrid models predict meaning-bearing tokens better, while transformers excel at verbatim repetition and structural elements.
• Encoder-Only Transformers (like BERT) for RAG, Clearly Explained!!! — Encoder-only Transformers create context-aware embeddings by integrating word embeddings, positional encoding, and self-attention.
• Stanford AI Lab Papers and Talks at ACL 2022 — ACL 2022 showcased diverse NLP research from Stanford, spanning model pretraining, analysis, and evaluation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.