Nobody gets this right

2026-06-07 · Source: David Shapiro · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, long

Summary

The article challenges the common misconception that language models (LLMs) cannot function as "world models" because "the world isn't made of words." The author argues this view is outdated, emphasizing that contemporary LLMs are increasingly multimodal, trained on diverse data including audio, video, images, and text, making "omni models" a more accurate descriptor. Several claims against LLMs as world models are refuted, such as the unpredictability of sensor data, the inability to predict pixels like tokens, and the semantic distinction between "generation" and "understanding" in action-conditioned models. The author also dismisses the idea that future AI companies will exclusively train world models on sensor data, pointing out that cognitive architectures for integrating multiple data streams have existed since the 1970s for autonomous systems like rockets.

Key takeaway

For AI architects and machine learning engineers evaluating advanced AI capabilities, recognize that the "world model" debate is evolving. Modern "omni models" integrate multimodal data, challenging the words-only limitation of traditional language models. Focus on an AI's predictive accuracy across diverse data types, as this demonstrates abstract understanding. Avoid outdated distinctions between generation and understanding, and consider established cognitive architecture principles for unifying complex sensor inputs in autonomous systems.

Key insights

The distinction between language models and world models is diminishing as AI becomes multimodal and adept at abstract mathematical representations.

Principles

• Modern LLMs are multimodal "omni models."
• Prediction accuracy indicates abstract understanding.
• Cognitive architectures unify diverse data streams.

In practice

• Consider multimodal AI for complex tasks.
• Evaluate AI by predictive accuracy, not input type.
• Integrate diverse sensor data using cognitive architectures.

Topics

• World Models
• Multimodal AI
• Omni Models
• Cognitive Architectures
• Generative AI
• Sensor Data Integration

Best for: AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• It's Tokens all the Way Down — Generative AI is unifying across modalities through a token-based, sequence-prediction architecture.
• Context Unrolling in Omni Models — Unified multimodal training enables "Context Unrolling" for enhanced cross-modal reasoning and performance.
• Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion — Mind-Omni unifies brain-vision-language tasks via discrete diffusion and a Brain Tokenizer for multi-modal interaction.
• Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages — Multimodal LLMs must integrate text, speech, and vision for low-resource languages, moving beyond English-centric benchmarks.
• What is Multimodal AI? How LLMs Process Text, Images, and More — Native multimodality uses a shared vector space for unified processing of diverse data types, enabling richer AI reasoning.
• Cosmos 3: Omnimodal World Models for Physical AI — Cosmos 3 unifies diverse modalities into a single world model, setting new performance benchmarks for Physical AI.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by David Shapiro.