Treating Decoder-Only LLMs as Encoders: A Simple and Effective Fine-tuning Approach for Named Entity Recognition

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Health & Medical Research · Depth: Expert, medium

Summary

A novel fine-tuning approach for decoder-only Large Language Models (LLMs) addresses their limitations in Named Entity Recognition (NER) by enabling full contextual understanding. Traditional decoder-only LLMs use causal masking, which restricts token representations from accessing right-side context, hindering performance on structured prediction tasks like NER. Researchers propose replacing causal attention with full attention during fine-tuning, coupled with label-supervised discriminative training. This method was rigorously evaluated on seven LLMs from four families: Gemma, Qwen2.5, Llama3.1, and Llama3.2. The findings demonstrate that the proposed technique, especially when configured with LoRA, surpasses established encoder baselines such as BERT, RoBERTa, and DeBERTa. It achieves robust NER performance without requiring auxiliary data or architectural changes, though it does not attain state-of-the-art results on BC5CDR and CoNLL2003 datasets.

Key takeaway

For Machine Learning Engineers aiming to enhance Named Entity Recognition (NER) performance using Large Language Models, you should consider fine-tuning decoder-only LLMs with full attention. This approach effectively overcomes the causal masking inherent in these models, enabling them to leverage full contextual information for token-level classification. Implement LoRA with this method to achieve strong results comparable to or exceeding traditional encoder baselines like BERT, without needing architectural changes or auxiliary data.

Key insights

Fine-tuning decoder-only LLMs with full attention enables strong NER performance by overcoming causal masking limitations.

Principles

• Causal masking limits decoder-only LLMs for token-level classification.
• Full attention fine-tuning can adapt LLMs for encoder-like tasks.
• LoRA can optimize performance for this adaptation.

Method

Fine-tune decoder-only LLMs by replacing causal attention with full attention. Combine this with label-supervised discriminative training. Evaluate with LoRA configurations.

In practice

• Adapt decoder LLMs for NER using full attention fine-tuning.
• Apply LoRA for efficient fine-tuning of adapted LLMs.
• Consider this method for structured prediction tasks.

Topics

• Named Entity Recognition
• Decoder-Only LLMs
• Fine-tuning
• Full Attention
• LoRA
• Structured Prediction

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats — Lightweight LLMs offer competitive biomedical NER performance, with specific output formats proving more effective than multi-format tuning.
• System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5 — Domain-specific datasets and LoRA fine-tuning significantly enhance LLM performance in specialized tasks like classical poetry analysis.
• Building an LLM From Scratch: The Mechanism That Changed AI Forever, Implemented From Zero — Self-attention enables LLMs to understand word relevance by dynamically weighting contextual dependencies.
• Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices — Complementary techniques enable LoRA fine-tuning of large LLMs on edge devices by drastically reducing peak memory without quality loss.
• Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting — LoRA fine-tuning can cause significant forgetting, which a new output space regularizer effectively mitigates without replay data.
• You Only Index Once: Cross-Layer Sparse Attention with Shared Routing — Cross-layer sparse attention (CLSA) shares KV cache and routing indices across layers to improve LLM inference efficiency and quality.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.