Step by step thinking to design Memory in sequential Neural Networks (RNN) — Part 2

2026-07-26 · Source: Deep Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

Part 2 of a series on designing memory in sequential neural networks (RNNs) details the transition from a scalar-weighted accumulation to matrix transformations. While Part 1's scalar weight encoded word order, it failed to learn relationships between words or decouple memory size from vocabulary size. This installment introduces replacing the scalar with `W_memory` and `W_input` matrices, allowing the network to combine old memory dimensions into new features (e.g., "Edibility" from "Fruitness"). Crucially, `W_input` transforms large one-hot vectors (e.g., 1,000,000 dimensions) into compact, fixed-size representations (e.g., 256 dimensions), effectively creating learned word embeddings. The `tanh` activation introduces non-linearity, and the RNN's hidden state is conceptualized as an evolving coordinate in a learned feature space, continuously rewriting the past into a representation useful for predicting the future.

Key takeaway

For Machine Learning Engineers designing or optimizing sequential models, understanding the shift from scalar to matrix-based memory updates in RNNs is critical. This enables the network to learn complex feature relationships and manage memory size efficiently, moving beyond simple positional encoding. Focus on how `W_memory` and `W_input` dynamically transform representations for predictive tasks.

Key insights

RNNs use matrix transformations to dynamically rewrite memory, enabling new feature combinations and managing vocabulary size efficiently.

Principles

• Matrices rewrite representations; scalars only scale them.
• RNN memory evolves as a coordinate in a learned feature space.
• RNNs prioritize information useful for prediction, discarding irrelevant data.

Method

The core RNN update `hₜ = tanh(Wₕhₜ₋₁ + Wₓxₜ + b)` transforms previous memory and current input using learned matrices, bias, and non-linear activation.

In practice

• Map large vocabulary one-hot vectors into compact embeddings via `W_input`.
• Set memory size independently of vocabulary size, e.g., 256 dimensions.

Topics

• Recurrent Neural Networks
• Neural Network Memory
• Word Embeddings
• Matrix Transformations
• Sequential Models
• Feature Learning

Best for: Machine Learning Engineer, AI Scientist, AI Student

Related on AIssential

• Step by step thinking to design Memory in sequential Neural Networks (RNN) — Part 1 — Effective sequential memory in NNs requires fixed-size, order-preserving encoding that learns meaning without pre-injected relationships.
• The Evolution of Language Modeling — Recurrent Neural Networks process variable-length sequential data using feedback loops and shared weights.
• recurrent neural networks (rnn) - explained #maths #dataanlysis #datascience #machinelearning — RNNs use a hidden state as a compressed memory, combining current input with past information.
• Memory Caching: RNNs with Growing Memory — Memory Caching enhances RNNs by enabling memory growth with sequence length, bridging the performance gap with Transformers.
• From RNNs to Transformers: How Sequence Models Evolved — Sequence models evolved by overcoming sequential processing and fixed context bottlenecks through dynamic attention and parallel computation.
• Learning to Recall with Transformers Beyond Orthogonal Embeddings — Transformer storage capacity scales multiplicatively with sample size, embedding dimension, and sequence length under random embeddings.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Deep Learning on Medium.