Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

2026-07-08 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Compact world models designed to ground spatial relations using language goals, such as "put the red block left of the blue block," are susceptible to "instruction leakage." Researchers found that a goal-conditioned predictor achieving 0.90 relation-readout accuracy often relies on instruction transcription rather than genuine perception. Evidence includes accuracy collapsing to 0.27 when the goal is withheld, and 94.5% of predictions following false instructions compared to 2.3% for the true scene. This leakage occurs when the scored quantity is directly transcribable from the instruction, independent of other inputs. While tabletop and BabyAI benchmarks exhibit this, a Language-Table forward-dynamics model only leaks if the instruction names the direction. The proposed fix involves removing the goal from the model's dynamics, treating it as a planner's cost, and supervising the "read" path, which restores instruction-independent grounding to 0.88. This detection and remedy protocol is applicable to any goal-conditioned world model where the instruction names the scored quantity.

Key takeaway

For AI Scientists and Machine Learning Engineers developing goal-conditioned world models, you must rigorously test for instruction leakage. If your model's high accuracy on spatial relations like "left of" stems from transcribing the instruction rather than genuine perception, your system lacks robust grounding. You should implement the proposed fix: keep the language goal out of the model's dynamics, treating it solely as a planner's cost, and focus supervision on the "read" path to ensure instruction-independent grounding.

Key insights

Goal-conditioned world models can suffer from "instruction leakage," where high accuracy stems from transcribing instructions, not true perception.

Principles

• Instruction leakage occurs when the instruction names the answer.
• Leakage is independent of non-instruction input predictiveness.
• The goal should be a planner's cost, not in dynamics.

Method

Detect leakage by testing goal-conditioned models with and without the goal, and with counterfactual instructions. Remedy by removing the goal from dynamics and supervising the "read" path.

In practice

• Test goal-conditioned models for instruction transcription.
• Isolate the language goal from model dynamics.
• Supervise the "read" path for genuine grounding.

Topics

• World Models
• Spatial Relations
• Instruction Leakage
• Language Grounding
• Model Robustness
• Goal-Conditioned AI

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• The Sequence Knowledge #800: Not All World Models are Created Equal — Model-Free RL agents learn reactive behaviors without internal environmental understanding.
• Pitwall: Faithful Natural-Language Race-Strategy Briefings from a Calibrated Real-Time Monte Carlo Engine — Faithfulness in grounded generation requires architectural verification and careful base model selection, not just data gating.
• ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop — Embodied spatial intelligence requires active perception-action loops, where agents actively seek information rather than passively processing it.
• From Prompts to Pavement Through Time: Temporal Grounding in Agentic Scene-to-Plan Reasoning — Temporal grounding in AV LLM/LMM ensembles reshapes reasoning but doesn't statistically improve standard correctness metrics, yet shows qualitative benefits.
• Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model — Structured action modeling and semantic grounding are critical for high-DoF world models to achieve fidelity and realism.
• Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling — Hamiltonian World Models offer a physically grounded approach to improve world model reliability and action utility.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.