LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens

2026-02-16 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, long

Summary

LLaMo is a novel unified framework developed by researchers from Brown University and Meta that extends pretrained Large Language Models (LLMs) for both human motion understanding and generation. It addresses critical challenges in existing approaches, such as catastrophic forgetting of linguistic capabilities due to limited motion-text data and jitter artifacts from discrete motion tokenization. LLaMo employs a modality-specific Mixture-of-Transformers (MoT) architecture, which preserves the base LLM's language understanding while enabling scalable multimodal adaptation. It encodes human motion into a causal continuous latent space and uses a lightweight flow-matching head for real-time streaming motion generation at $\geq$30 FPS. The model was pretrained on a new in-house dataset of over 3 million motion sequences (3,076 hours) and demonstrates high-fidelity text-to-motion generation and motion-to-text captioning, including strong zero-shot performance, without compromising the LLM's original text-only capabilities.

Key takeaway

For Research Scientists developing multimodal AI, LLaMo demonstrates a robust method to integrate human motion capabilities into existing LLMs without degrading their core language performance. You should consider adopting modality-specific architectures and continuous latent spaces for new modalities to avoid catastrophic forgetting and quantization artifacts, especially when aiming for real-time, high-fidelity generation in resource-constrained environments.

Key insights

LLaMo unifies motion-language understanding and generation in LLMs using continuous tokens and a MoT architecture.

Principles

• Preserve base LLM language competence.
• Avoid discrete tokenization for continuous data.
• Enable real-time streaming generation.

Method

LLaMo uses a MoT architecture to separate motion and language parameters, a causal continuous latent space for motion, and a flow-matching head for autoregressive generation, all built on a decoder-only Transformer backbone.

In practice

• Integrate motion generation into LLMs.
• Develop real-time motion synthesis systems.
• Improve zero-shot motion generation.

Topics

• LLaMo
• Motion-Language Models
• Continuous Motion Generation
• Mixture-of-Transformers
• Unified Multimodal AI

Best for: Research Scientist, AI Researcher, AI Scientist, Deep Learning Engineer

Related on AIssential

• AI News: This Video Model Has Everyone Freaked Out! — New AI models are pushing boundaries in multimodal video generation, accelerated code synthesis, and cost-effective, agentic LLMs.
• Social Structure Matters in 3D Human-Human Interaction Generation — LLMs can plan social structure for 3D human-human interaction, but require a separate motion executor to generate physically plausible movements.
• My Deep Dive into Large Language Models: An Architectural Journey — The shift to LLMs redefines NLP through scalable Transformer architectures, enabling versatile language tasks and requiring advanced customization.
• Why Vision Language Models Ignore What They See [Munawar Hayat] - 758 — Current VLMs struggle with physics-based generation, visual grounding, and multi-person image fidelity, necessitating improved training and architectural approaches.
• Claude Fable 5 Drops (Beats Opus) — Multimodal language models unify diverse data streams by tokenizing all modalities for transformer-based auto-regressive generation.
• IEEE Transactions on Artificial Intelligence, Volume 7, Issue 2, February 2026 — The volume showcases diverse AI advancements, from LLM applications in autonomous driving to specialized models for protein language processing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.