On the Expressive Power and Limitations of Multi-Layer SSMs

2026-04-16 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A recent study investigates the expressive power and inherent limitations of multi-layer state-space models (SSMs), particularly in compositional tasks, identifying a fundamental gap between SSMs and streaming models. The research explores the impact of Chain-of-Thought (CoT) reasoning, demonstrating that offline CoT does not enhance expressiveness, whereas online CoT significantly boosts SSM capabilities, making them equivalent in power to streaming algorithms. Furthermore, the study analyzes the trade-off between model width and precision, concluding that these resources are not interchangeable in base SSMs but achieve equivalence when online CoT is incorporated. These findings provide a unified understanding of how depth, finite precision, and CoT influence the performance boundaries of SSMs.

Key takeaway

For AI scientists developing or deploying state-space models, understanding that online Chain-of-Thought (CoT) reasoning can bridge the expressiveness gap with streaming algorithms is crucial. You should prioritize integrating online CoT into your multi-layer SSM architectures, especially for compositional tasks, to overcome inherent limitations and achieve greater computational power. This approach also clarifies resource allocation, showing that width and precision become equivalent with online CoT.

Key insights

Multi-layer SSMs have inherent compositional limitations, but online Chain-of-Thought reasoning can make them as powerful as streaming algorithms.

Principles

• SSMs inherently struggle with compositional tasks.
• Online CoT significantly enhances SSM expressiveness.
• Width and precision are not interchangeable in base SSMs.

In practice

• Implement online CoT for complex SSM tasks.
• Prioritize online CoT over offline for expressiveness.

Topics

• Multi-layer SSMs
• Expressive Power
• Computational Limitations
• Chain-of-Thought
• Streaming Algorithms

Best for: AI Scientist, Research Scientist

Related on AIssential

• On the Expressive Power and Limitations of Multi-Layer SSMs — Online chain-of-thought significantly boosts multi-layer SSM expressiveness, bridging the gap with streaming algorithms.
• Efficiently Representing Algorithms With Chain-of-Thought Transformers — CoT transformers can efficiently simulate Word RAM algorithms with poly-logarithmic overhead, surpassing Turing machine efficiency.
• CoT-Space: A Theoretical Framework for Internal Slow-Thinking via Reinforcement Learning — CoT-Space reframes LLM reasoning as continuous optimization in a semantic space, explaining optimal CoT length.
• Feedback request: Testing the $H_{dp}$ bandwidth bound on LLM benchmarks (Preprint check & review) — The $H_{dp}$ bandwidth bound predicts CoT benefits only when task depth exceeds a transformer's single-pass capacity.
• LLMs+Graphs: Toward Graph-Native, Synergistic AI Systems — LLMs require graph integration to overcome structured reasoning limitations and enable graph-native AI systems.
• Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations — LLM robustness to CoT perturbations varies significantly by type and model scale, challenging assumptions that size alone ensures reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.