Sampling from Flow Language Models via Marginal-Conditioned Bridges

2026-05-14 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

Flow Language Models (FLMs) are a new class of language models that apply continuous flow matching to one-hot encoded token sequences. Unlike generic continuous diffusion models, FLMs have a special denoiser structure where each block of the denoising mean represents a posterior marginal distribution over the clean token at that position. Standard DDPM-style samplers collapse these marginals to a single conditional-mean endpoint, which is not a valid one-hot sequence. This work introduces a training-free marginal-conditioned bridge (MCB) sampler for FLMs. The MCB sampler samples a clean one-hot endpoint from the factorized posterior defined by FLM token marginals and then samples the next continuous state from an analytic Ornstein–Uhlenbeck bridge conditioned on that endpoint. This method uses the same model evaluations as standard sampling and provides a principled interface for token-level decoding controls like temperature scaling and nucleus truncation. Experiments on LM1B show that the MCB sampler improves the quality–diversity tradeoff compared to standard ODE samplers.

Key takeaway

For research scientists developing or deploying Flow Language Models, adopting the marginal-conditioned bridge (MCB) sampler is crucial. This training-free method directly leverages FLM's inherent posterior marginals, offering superior control over generation quality and diversity via standard decoding techniques like temperature scaling and nucleus sampling. You can achieve better generative perplexity and maintain non-collapsed entropy, even with fewer sampling steps, without additional model evaluations.

Key insights

FLMs' token-wise posterior marginals enable a more principled, training-free sampling method for improved text generation.

Principles

• FLM denoisers provide token-wise posterior marginals.
• Sampling from posterior marginals improves quality-diversity tradeoff.
• Girsanov path-space comparison shows denoising error reduction.

Method

The MCB sampler samples a clean one-hot endpoint from factorized posterior marginals, then samples the next continuous state from an analytic Ornstein–Uhlenbeck bridge conditioned on that endpoint.

In practice

• Apply temperature scaling for quality-diversity control.
• Use nucleus sampling (e.g., p=0.95) for additional improvement.
• Integrate into existing FLM workflows without retraining.

Topics

• Flow Language Models
• Marginal-Conditioned Bridges
• Non-autoregressive Language Models
• Ornstein–Uhlenbeck Bridge
• Text Generation

Code references

• imbirik/mcb

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Mean-Field Parallel Decoding for Discrete Diffusion Language Models — A training-free framework coordinates parallel token updates in discrete diffusion models, improving generation quality and latency.
• Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification — TTRL-CoCoV improves LLM reasoning by confidence-conditioned verification, addressing pseudo-label errors and diversity collapse in label-free settings.
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data — Structured pedagogical formats and diverse outputs are key to high-quality synthetic pretraining data, not larger generator models.
• Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching — MoE-FM enables faster, high-quality non-autoregressive language model inference by handling complex latent distributions.
• DALM: A Domain-Algebraic Language Model via Three-Phase Structured Generation — DALM reframes language generation as algebraically constrained structured denoising over a domain lattice.
• SAID: Accelerating Diffusion-Based Language Models via Scaffold-Aware Iterative Decoding — SAID accelerates DLLMs by prioritizing denoising on structural tokens and adaptively processing uncertain tokens.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.