Seeking collaborators for scaling and independent evaluation of a new recurrent language model architecture (preprint + code) [R]

2026-07-16 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, short

Summary

A new recurrent language model architecture, DABSN (Dynamic Adaptive Bias State Network), has been introduced via a preprint and public code (PyTorch, C++, Triton). The initial paper details DABSN's behavior on reasoning, memory, and long-sequence benchmarks like MQAR, Copy, Key-Value retrieval, and A5/60. The developer also trained a 24M parameter language model using 1B pretraining tokens and a GPT-2 tokenizer, yielding promising results. A second paper is planned to focus on language modeling, long-context capabilities, and scaling. The project is open and reproducible, and the developer is actively seeking collaborators for independent reproduction, baseline design, and access to larger GPU clusters to scale the architecture further. DABSN addresses traditional RNN parallelism issues through a diagonal scan, akin to modern state space models.

Key takeaway

For AI researchers or machine learning engineers exploring novel language model architectures, consider evaluating DABSN. Its diagonal scan approach addresses RNN parallelism challenges, potentially offering an alternative to transformer or state space models for long-context tasks. You should investigate its open-source code and contribute to its independent evaluation and scaling efforts, particularly by designing robust benchmarks that test generalization beyond training data and "reach tests" with confounding information.

Key insights

A new recurrent neural network architecture, DABSN, offers parallel processing capabilities similar to state space models, addressing traditional RNN limitations.

Principles

• Recurrent architectures can achieve parallelism through diagonal scans.
• Language model evaluation requires robust generalization beyond training data.
• Open collaboration accelerates architecture scaling and independent validation.

Method

The DABSN architecture is implemented with PyTorch, C++, and Triton, and evaluated on reasoning, memory, and long-sequence benchmarks.

In practice

• Evaluate language models for x10 generalization beyond training data.
• Implement "reach tests" with confounding data to assess long-context robustness.
• Explore diagonal scan implementations for recurrent model parallelism.

Topics

• Recurrent Neural Networks
• Language Models
• DABSN Architecture
• Model Scaling
• Long-Context Benchmarks
• Open Collaboration
• PyTorch

Code references

• BleedingXiko/dabsn

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Introduction to RNNs — RNNs use a self-loop and hidden state to process sequential data, maintaining context over time.
• A Dual-Path Architecture for Scaling Compute and Capacity in LLMs — A dual-path architecture scales LLM compute and capacity simultaneously using deep and wide sublayers with per-token gates.
• Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training — Forward-Forward learning's real-data scaling and memory efficiency are significantly overstated by synthetic benchmarks, revealing limitations compared to backpropagation.
• ParaRNN: Large-Scale Nonlinear RNNs, Trainable in Parallel — ParaRNN enables parallel training of large-scale nonlinear RNNs, matching transformer performance with superior inference efficiency.
• Looped State-Space Language Models with Adaptive Exit-State Selection — Looped state-space models, like Looped Mamba, achieve computational depth through recurrent application of shared blocks, outperforming non-looped baselines with fewer parameters.
• Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence — Structured recurrent states can be parameter-efficient for language modeling but require explicit sparse retrieval for exact long-range recall.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.