Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

2026-06-08 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

Together AI's research project, "Road to 5 Million Sequence Length," addresses memory barriers in training large language models with extended context. The project successfully scaled transformer-based models, like a Llama 3B architecture, to 5 million tokens on an 8x H100 GPU node. Key bottlenecks include quadratic computation and linear memory growth. The team combined existing techniques such as Fully Sharded Data Parallelism (FSDP), DeepSpeed Ulysses context parallelism, activation checkpointing, and offloading activations to CPU. They also introduced "Arctic sequence length training" and a novel optimization called "Untitled Ulysses," which further refines context parallelism by chunking attention head computations to reuse buffers and reduce activation memory without significant throughput impact. This approach enables training at unprecedented context lengths.

Key takeaway

For AI Architects and ML Engineers designing or fine-tuning large language models for agentic or video generation applications, understanding and implementing advanced memory optimization techniques is crucial. You should combine sharded data parallelism, context parallelism like DeepSpeed Ulysses, activation checkpointing, and CPU offloading. Consider Together AI's "Untitled Ulysses" approach to push context lengths beyond 3 million tokens, enabling more efficient use of GPU resources for extreme sequence lengths.

Key insights

Achieving 5 million token context in LLMs requires stacking multiple memory optimization techniques.

Principles

• Memory bottlenecks appear unexpectedly in long context training.
• Context parallelism can be further optimized by chunking attention heads.
• Recomputing activations reduces memory footprint significantly.

Method

Combine FSDP, DeepSpeed Ulysses, activation checkpointing, CPU offloading for activations, Arctic sequence length training, and "Untitled Ulysses" for chunked attention head computation.

In practice

• Use PyTorch profiler to identify memory bottlenecks.
• Implement DeepSpeed Ulysses for attention parallelism.
• Offload non-critical activations to CPU.

Topics

• Long Context LLMs
• Context Parallelism
• DeepSpeed Ulysses
• Activation Checkpointing
• Memory Optimization
• Together AI
• Transformer Models

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Ulysses Sequence Parallelism: Training with Million-Token Contexts — Ulysses SP enables training LLMs with million-token contexts by distributing attention heads across GPUs, significantly reducing memory and boosting throughput.
• The Sequence AI of the Week #851: DeepSeek-V4 and the Architecture of Million-Token Intelligence — Effective long-context reasoning requires a holistic systems approach beyond just scaling Transformer models.
• Latent Context Compilation: Distilling Long Context into Compact Portable Memory — Latent Context Compilation distills long LLM contexts into portable buffer tokens using a disposable LoRA and self-aligned optimization.
• Friends Don’t Let Friends Run Loops Sequentially — Parallel processing significantly accelerates I/O-bound LLM inference by issuing concurrent requests, reducing wall-clock time.
• Data Machina #262 — New models and frameworks are advancing LLM efficiency, reasoning, and long-form content generation.
• Learning When to Attend: Conditional Memory Access for Long-Context LLMs — Conditional attention mechanisms can significantly extend LLM context windows while improving efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.