New Server Hopes to Break Through AI’s “Memory Wall”

2026-06-01 · Source: IEEE Spectrum · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, quick

Summary

Majestic Labs, an AI hardware startup, is developing Prometheus, a new AI server designed to overcome the "memory wall" bottleneck in large language model (LLM) inference. This server will feature up to 128 terabytes of memory, over 60 times more than Nvidia's DGX B300 server. Prometheus employs a DRAM-centric architecture, utilizing LPDDR6 and a proprietary memory interface with miniature copper cables to achieve memory bandwidth up to 25.6 terabytes per second. Its compute engine is the custom Ignite AI processing unit, which integrates ARM application cores with RISC-V vector and tensor cores on a single die, with 12 Ignite chips per server. Prometheus will support PyTorch, vLLM, and OpenAI's Triton inference frameworks without code modifications. The Open Compute Project-compliant server, expected to ship in 2027, will support modular memory upgrades, up to 120 kilowatts per rack, and cold-plate liquid cooling. Majestic Labs claims this design will reduce capital expenditure and power consumption by 10 to 50 times compared to current solutions.

Key takeaway

For AI Architects and Machine Learning Engineers designing LLM inference infrastructure, Majestic Labs' Prometheus server presents a significant shift. If you are struggling with memory bottlenecks and high operational costs, consider how a DRAM-centric, high-capacity server could drastically reduce capital expenditure and power consumption. You should evaluate this architecture's potential to scale LLM inference more economically than current HBM-limited systems, especially as models continue to grow.

Key insights

Majestic Labs' Prometheus server aims to break the LLM memory wall with a DRAM-centric architecture and integrated AI processing.

Principles

• LLM token generation is memory-bound.
• HBM limits memory capacity due to interface design.
• Unified DRAM architecture can scale memory.

Method

Majestic Labs' Prometheus server uses a proprietary copper cable memory interface and custom aggregation chips to scale LPDDR6 DRAM, paired with Ignite AI processors for unified compute.

In practice

• Consider DRAM-centric designs for LLM memory scaling.
• Explore unified compute/memory architectures for inference.
• Evaluate systems supporting existing AI frameworks (PyTorch, vLLM, Triton).

Topics

• AI Hardware
• LLM Inference
• DRAM Architecture
• Prometheus Server
• Ignite Processor
• Memory Wall

Best for: AI Engineer, NLP Engineer, AI Hardware Engineer, AI Architect, Machine Learning Engineer

Related on AIssential

• Majestic Labs Raises $100M for Memory Pooling AI Server — Majestic Labs' memory-pooled server disaggregates memory from compute via a high-bandwidth interface, providing vast, unified DRAM for AI inference.
• A Practical Guide to CPU-Optimized LLM Deployment on Intel® Xeon® 6 Processors on AWS. — Intel Xeon 6 processors with vLLM enable cost-effective, high-throughput LLM inference on CPUs.
• Breaking the Memory Wall in the Age of Inference — Integrating compute directly into SRAM cells via Digital In-Memory Compute (DIMC) effectively addresses AI inference memory bottlenecks.
• Taalas Specializes to Extremes for Extraordinary Token Speed — Extreme specialization in AI chips by hardwiring models can yield superior performance and cost efficiency.
• Run Frontier AI at Home — Alex Cheema, EXO Labs — Exo Labs enables efficient local frontier AI inference by optimizing the full stack across heterogeneous hardware.
• This chip startup just raised $135M on a bet that AI’s biggest bottleneck isn’t compute — it’s memory — AI inference is increasingly a memory scaling problem, not solely a compute challenge.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by IEEE Spectrum.