Sumi: Open Uniform Diffusion Language Model from Scratch

2026-06-17 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Sumi is introduced as a fully open 7B uniform diffusion language model, pretrained from scratch on 1.5 trillion tokens. This release addresses a gap in the field, as no uniform diffusion language model (UDLM) has previously been pretrained at both large parameter scale and large token budget, unlike autoregressive and masked diffusion models. Sumi performs competitively with autoregressive models on knowledge, reasoning, and coding benchmarks, though it underperforms on commonsense tasks, potentially due to its education-heavy data mixture. The project aims to provide a clean reference point for studying UDLM scaling behavior, generation dynamics, controllability, and trade-offs, with its model weights, checkpoints, and full training recipe publicly available.

Key takeaway

For AI scientists and machine learning engineers exploring novel language model architectures, Sumi provides a critical open-source baseline for uniform diffusion. You can now directly study its scaling behavior, generation dynamics, and controllability against established autoregressive models, leveraging its released weights and training recipe. This enables deeper research into the unique properties and potential of uniform diffusion, informing future model development and architectural choices.

Key insights

Sumi is the first large-scale, scratch-pretrained uniform diffusion language model, offering a new open research baseline.

Principles

• Uniform diffusion models allow flexible token updates.
• Large-scale scratch pretraining provides clean research baselines.
• Data mixture significantly influences model performance.

Method

Sumi was pretrained from scratch as a 7B uniform diffusion language model on 1.5T tokens, utilizing a specified data mixture from publicly available corpora.

In practice

• Study native uniform diffusion at scale.
• Analyze generation dynamics and controllability.
• Compare UDLM trade-offs against autoregressive models.

Topics

• Uniform Diffusion Models
• Language Models
• Model Pretraining
• Open-Source AI
• Scaling Laws
• Benchmarking

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• TiDAR: Think in Diffusion, Talk in Autoregression (Paper Analysis) — TiDAR accelerates LLM inference by leveraging unused GPU capacity for parallel diffusion-based token drafting and autoregressive validation.
• Latest open artifacts (#18): Arcee's 400B MoE, LiquidAI's underrated 1B model, new Kimi, and anticipation of a busy month — Smaller open models are achieving competitive performance against larger counterparts through extensive pretraining and architectural innovations.
• Detect, Remask, Repair: Diffusion Editing for Faithful Summarization of Evolving Contexts — Localized diffusion-based repair updates outdated summary spans, preserving supported content more efficiently than full regeneration.
• Mean-Field Parallel Decoding for Discrete Diffusion Language Models — A training-free framework coordinates parallel token updates in discrete diffusion models, improving generation quality and latency.
• Towards Speed-of-Light Text Generation with Nemotron-Labs Diffusion Language Models — Diffusion Language Models (DLMs) enable parallel, iterative text generation and revision, significantly boosting speed and accuracy over autoregressive models.
• Teaching Diffusion to Speculate Left-to-Right — Aligning bidirectional diffusion drafters with left-to-right autoregressive verification significantly boosts speculative decoding efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.