ModernBERT - Modern Replacement for BERT | RAG, Embeddings, Classification, Reranking

2026-03-08 · Source: Venelin Valkov · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Advanced, long

Summary

ModernBERT, introduced in December 2024 by Answer AI, White on AI, and Nvidia researchers, represents a significant advancement in encoder-only Transformer models, addressing a gap in improvements for this architecture. It replaces absolute positional encodings with RoPE embeddings, enabling wider context windows up to 8K tokens, and incorporates changes in normalization and activation functions. A key innovation is "alternating attention," which applies global attention every third layer and local attention (128 tokens) on others, making it compute-efficient with linear time complexity. ModernBERT was trained on over 2 trillion tokens, a substantial increase from the original BERT's 3.3 billion, primarily English text with code and mathematical equations. Training involved three phases, increasing masked tokens to 30%, and removing the next sentence prediction objective. A multilingual extension, MMERT, supports over 1,800 languages using 3 trillion tokens and the Gemma 2 multilingual tokenizer.

Key takeaway

For AI Engineers working with natural language processing tasks, ModernBERT offers a compelling upgrade over previous encoder-only models. Its enhanced context window (up to 8K tokens) and superior inference speed make it ideal for production environments requiring efficient text classification, embedding generation, or retrieval-augmented generation. You should consider integrating ModernBERT into your pipelines, especially for applications demanding high performance and broader contextual understanding, or its multilingual variant for global deployments.

Key insights

ModernBERT significantly advances encoder-only models with architectural innovations and massive training data, improving performance and context.

Principles

• Alternating attention enhances context window efficiency.
• Large-scale, diverse training data improves model capabilities.
• Progressive training phases optimize model learning.

Method

ModernBERT employs RoPE embeddings, alternating attention (global/local), and a three-phase training regimen with increased masked tokens (30%) and sequence packing for efficiency.

In practice

• Use ModernBERT for classification, entity recognition, and embeddings.
• Leverage Hugging Face Transformers for easy loading and fine-tuning.
• Explore MMERT for multilingual applications.

Topics

• ModernBERT
• Transformer Models
• Encoder-Decoder Architectures
• Masked Language Modeling
• Multilingual Models

Best for: AI Engineer, Machine Learning Engineer, Deep Learning Engineer

Related on AIssential

• NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus — NorBERTo is a new ModernBERT encoder for Portuguese, trained on the massive Aurora-PT corpus, showing strong benchmark performance.
• Introducing the Ettin Reranker Family — Ettin Rerankers offer state-of-the-art accuracy and speed across various sizes via distillation and optimized architecture.
• BERT in NLP: How an Encoder-Only Transformer Changed Language Understanding — BERT's encoder-only, bidirectional pretraining revolutionized NLP by enabling broad language understanding and easy task adaptation.
• LinkBERT: Improving Language Model Training with Document Link — Incorporating document links during pretraining significantly boosts language models' multi-hop reasoning and knowledge acquisition.
• ModernBERT is more efficient than conventional BERT for chest CT findings classification in Japanese radiology reports — ModernBERT offers computational efficiency for Japanese medical text classification but requires diverse data for robustness.
• Large Language Models Do Not Always Need Readable Language — LLMs can effectively process compact, non-human-readable "BabelTele" representations, preserving semantics and reducing context overhead.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Venelin Valkov.