[R] Tiny transformers (<100 params) can add two 10-digit numbers to 100% accuracy

2026-02-28 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, short

Summary

A project demonstrates that tiny transformer models, with fewer than 100 parameters, can achieve 100% accuracy in adding two 10-digit numbers. This performance is attributed to the use of digit tokens, simplifying the task compared to floating-point arithmetic. The research explores the minimal transformer architecture required for integer addition, highlighting that manually selecting weights can significantly reduce parameter counts compared to conventionally optimized models. This work suggests potential for shrinking models and understanding transformer internal mechanisms, particularly in how they learn simple, rule-based operations.

Key takeaway

For research scientists exploring model efficiency and interpretability, this work suggests that highly specialized, minimal transformer architectures can achieve perfect accuracy on specific tasks. You should consider how manual weight selection or task-specific tokenization might inform the design of more efficient models, potentially reducing the need for extensive training data and compute budgets in certain problem domains.

Key insights

Tiny transformers can achieve perfect accuracy on 10-digit addition using minimal parameters.

Principles

• Manual weight selection can drastically reduce parameters.
• Digit tokenization simplifies arithmetic tasks for models.

Method

The project focuses on finding the minimal transformer architecture capable of representing integer addition, leveraging digit tokens and potentially hand-picked weights for efficiency.

In practice

• Explore digit tokenization for numeric tasks.
• Investigate manual weight initialization for minimal models.

Topics

• Tiny Transformers
• Model Compression
• Integer Addition
• Neural Network Architectures
• Lottery Ticket Hypothesis

Code references

• anadim/AdderBoard

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment — Lightweight Transformers can match traditional ML for on-device fault detection but demand significant resource tradeoffs, especially on imbalanced data.
• ARC Prize 2025 Paper Award 1st Place TRM — TRM demonstrates that small, recursive models can achieve strong generalization on tiny datasets by learning to iteratively improve answers.
• Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective — Transformers can overcome the curse of dimensionality, demonstrating strong expressive capabilities for function approximation.
• Transformers Step-by-Step Explained (Attention Is All You Need) — Transformers enable parallel processing and efficient context capture in sequential data through a dynamic attention mechanism.
• How to Optimize Transformer-Based Models for Low-Precision Training — Optimizing low-precision Transformer training requires benchmarking actual GEMM workloads to understand real speedups.
• The Discrete-Log Clock: How a Transformer Learns Modular Multiplication — The transformer learns modular multiplication by reducing it to addition in discrete-log space, revealed by using the correct multiplicative character transform.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.