TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models

2026-06-12 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

TimeROME-DLM is the first training-free, gradient-free, inference-time knowledge-editing framework specifically designed for masked diffusion language models (MDLMs). Existing knowledge-editing methods for autoregressive LLMs fail on MDLMs due to their iterative denoising process. TimeROME-DLM integrates a Temporal Indirect Effect (TIE) causal-tracing protocol to pinpoint the exact coordinate (layer, denoising step, module) where a fact is recoverable, and a closed-form, low-rank residual edit memory. This framework reduces forget-set log-probability by approximately 83 nats on TOFU forget01 with LLaDA-8B-Base, preserves retain-set log-probability within ~1 nat across 50 sequential insertions, and offers a four- to fourteen-fold wall-clock speedup with zero additional VRAM. It scales sub-linearly to 400 facts and transfers across six MDLM backbones, including LLaDA-8B-Instruct and Dream-7B.

Key takeaway

For AI Scientists and Machine Learning Engineers working with masked diffusion language models, you should consider TimeROME-DLM for knowledge editing and unlearning tasks. This framework enables efficient, gradient-free fact removal or modification at inference time, avoiding the high VRAM and computational costs of traditional gradient-based methods. Its ability to scale to hundreds of facts and maintain utility stability during sequential insertions makes it a robust solution for managing knowledge in MDLM deployments.

Key insights

MDLM knowledge editing requires tracing causality along the denoising trajectory, enabling gradient-free, inference-time updates.

Principles

• MDLM facts localize to the residual stream at early-middle denoising steps.
• Knowledge editing for MDLMs must follow the denoising trajectory.
• Multi-fact edits benefit from ridge-regularized least-squares for stability.

Method

TimeROME-DLM uses Temporal Indirect Effect (TIE) tracing to find the optimal (layer, denoising-step, module) coordinate. A low-rank residual update, aggregating all forget facts, is then applied at this coordinate during every diffusion forward pass.

In practice

• Apply edits at inference time to avoid VRAM costs and slow training.
• Use TIE tracing to pinpoint optimal intervention coordinates in MDLMs.
• Aggregate multiple forget facts into a single, stable low-rank update.

Topics

• Masked Diffusion Language Models
• Knowledge Editing
• Model Unlearning
• Causal Tracing
• Inference-Time Editing
• LLaDA

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models — TimeROME-DLM enables efficient, training-free knowledge editing for Masked Diffusion Language Models, overcoming limitations of AR LLM methods.
• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL [R] — MDLMs' any-order denoising objective enables globally coherent text generation, outperforming autoregressive models in agentic RL world modeling.
• DLM-SWAI: Steering Diffusion Language Models Before They Unmask — DLM-SWAI enables training-free steering of Diffusion Language Models by biasing token distributions during denoising.
• Machine Unlearning for Masked Diffusion Language Models — Masked Diffusion Unlearning (MDU) is the first framework for unlearning specific knowledge in Masked Diffusion Language Models.
• Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models — Attention-guided denoising and optimization (AGDO) significantly enhances dLLM reasoning by leveraging intrinsic token dependencies.
• One-step Language Modeling via Continuous Denoising — Continuous denoising via flow-based models can surpass discrete diffusion in language model quality and speed.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.