On the Expressive Power and Limitations of Multi-Layer SSMs

2026-04-16 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

A study by Nikola Zubić, Qian Li, Yuyi Wang, and Davide Scaramuzza investigates the expressive power and limitations of multi-layer state-space models (SSMs). The research reveals that multi-layer SSMs inherently face limitations in compositional tasks, creating a gap between SSMs and streaming models. However, the integration of online chain-of-thought (CoT) significantly enhances SSMs' capabilities, making them equivalent in power to streaming algorithms. The study also explores the interplay between model width and precision, demonstrating that these resources are not interchangeable in base SSMs but achieve equivalence when online CoT is employed. These findings provide a unified perspective on how depth, finite precision, and CoT influence the performance boundaries of SSMs.

Key takeaway

For research scientists developing or deploying state-space models, understanding the role of online chain-of-thought is critical. Implementing online CoT can overcome fundamental limitations in compositional tasks and achieve expressiveness comparable to streaming algorithms, potentially enabling new applications or improving existing ones where SSMs previously struggled. Consider integrating online CoT to enhance model capabilities and resource interchangeability.

Key insights

Online chain-of-thought significantly boosts multi-layer SSM expressiveness, bridging the gap with streaming algorithms.

Principles

• SSMs have inherent compositional task limitations.
• Online CoT can equalize SSMs with streaming algorithms.
• Width and precision are not interchangeable without online CoT.

Topics

• Multi-Layer SSMs
• Expressive Power
• Chain-of-Thought
• Streaming Algorithms
• Width-Precision Tradeoff

Code references

• she1110/FGPD
• enyac-group/Quamba

Best for: Research Scientist, AI Scientist, AI Student

Related on AIssential

• On the Expressive Power and Limitations of Multi-Layer SSMs — Multi-layer SSMs have inherent compositional limitations, but online Chain-of-Thought reasoning can make them as powerful as streaming algorithms.
• Scaling LLMs horizontally: hidden-state coupling without weight modification [R] — Residual Coupling enables horizontal scaling of frozen LLMs via linear bridges, enhancing performance without weight modification.
• S-Bus: Automatic Read-Set Reconstruction for Multi-Agent LLM State Coordination — S-Bus prevents multi-agent LLM structural race conditions via automatic HTTP-based read-set reconstruction and optimistic concurrency control.
• Feedback request: Testing the $H_{dp}$ bandwidth bound on LLM benchmarks (Preprint check & review) — The $H_{dp}$ bandwidth bound predicts CoT benefits only when task depth exceeds a transformer's single-pass capacity.
• The Sequence Knowledge #858: How State Space Models Went from Curiosity to Serious Transformer Competitor — State space models offer linear scaling and constant memory, challenging Transformers' quadratic complexity.
• Cost-Aware Model Orchestration for LLM-based Systems — Integrating quantitative performance and energy metrics into LLM orchestration significantly improves accuracy and efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.