It's Tokens all the Way Down

2025-09-10 · Source: The Computist Journal · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

Generative AI models, initially developed for separate modalities like text, images, and audio, are converging into a single token-based architecture. The core concept, often called a "language model," predicts the next token in a sequence, regardless of its origin. Historically, image generation evolved from GANs, which introduced the latent space concept, to diffusion models. Audio and text processing also had distinct approaches. A pivotal moment was OpenAI's 2021 CLIP model, which aligned text and image embeddings in a shared latent space, enabling text-steered image generation. By the mid-2020s, the prevailing method became tokenizing all inputs—image patches, audio chunks—and feeding them into a unified model that predicts the next token, effectively making the distinction between "language" and "image" models obsolete.

Key takeaway

For AI Engineers designing multimodal systems, recognize that the underlying architecture is converging to a unified token-prediction model. This means you should focus on effective tokenization strategies across modalities rather than separate, bespoke systems. Your efforts in aligning different data types into a common latent space will yield more general and efficient models. Consider how a single sequence-modeling approach simplifies development and deployment.

Key insights

Generative AI is unifying across modalities through a token-based, sequence-prediction architecture.

Principles

• Generative models learn data distributions to sample new examples.
• Any sequence of distinct symbols can be modeled as a language.
• Latent spaces provide compressed, continuous maps of data variations.

Method

Tokenize all modalities (text, image patches, audio chunks) into a single stream. Train a unified model to predict the next token in this combined sequence.

In practice

• Utilize CLIP-like models for cross-modal embedding alignment.
• Employ diffusion models for robust image generation from noise.

Topics

• Generative AI
• Multimodal AI
• Tokenization
• Large Language Models
• Diffusion Models
• CLIP Model

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification — A single discrete visual tokenizer unifies multimodal understanding and generation in autoregressive models.
• Nobody gets this right — The distinction between language models and world models is diminishing as AI becomes multimodal and adept at abstract mathematical representations.
• What is Multimodal AI? How LLMs Process Text, Images, and More — Native multimodality uses a shared vector space for unified processing of diverse data types, enabling richer AI reasoning.
• LatentUMM: Dual Latent Alignment for Unified Multimodal Models — LatentUMM aligns latent space transformations to resolve functional inconsistencies in unified multimodal models.
• HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers — HYDRA-X unifies image and video tokenization in a single ViT, improving multimodal model efficiency and semantic awareness.
• Your Chatbot Is Playing You — Unifying diverse modalities through tokenization and specialized Transformer architectures is key to advanced multimodal AI.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Computist Journal.