I built a compiler that turns computation graphs into the weights of a vanilla transformer — no training anywhere [P]

2026-07-24 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A new compiler, "torchwright", has been developed to convert computation graphs defined in ordinary Python directly into the weights of a vanilla transformer model. This tool generates a standard Phi-3-architecture checkpoint that can be loaded by Hugging Face without any custom code or "trust_remote_code", entirely bypassing the traditional training process. The project aims to investigate the inherent algorithmic expressiveness of transformers, separate from their learning capabilities. Unlike previous approaches like RASP and Tracr, which also compile programs into transformer weights, "torchwright" prioritizes using standard Python for graph definition and targeting a stock transformer architecture for seamless integration. The repository includes twelve runnable examples demonstrating its functionality.

Key takeaway

For Machine Learning Engineers exploring model efficiency or interpretability, this compiler offers a novel approach to embed deterministic computations directly into transformer architectures. You can now define specific algorithms in Python and compile them into Phi-3 weights, potentially reducing training overhead for certain tasks or integrating operations that are difficult for neural networks to learn efficiently. Consider experimenting with "torchwright" to inject "frozen tools" or complex, non-learnable functions into your models, enhancing their capabilities without extensive retraining.

Key insights

A compiler directly generates vanilla transformer weights from Python computation graphs, demonstrating inherent algorithmic expressiveness without training.

Principles

• Transformers possess inherent algorithmic expressiveness.
• Manual design complements learned components in ML.
• Decoupling learned and deterministic ops aids analysis.

Method

Define a computation graph in ordinary Python, then compile it to generate weights for a standard Phi-3-architecture transformer, loadable directly via Hugging Face.

In practice

• Inject static, complex computations into networks.
• Integrate operations like square root directly into models.
• Hybridize custom-weight models with trained networks.

Topics

• Transformer Architecture
• Computation Graphs
• Model Compilation
• Phi-3 Model
• Hugging Face
• Untrained Models

Code references

• physicsrob/torchwright

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Stanford CS 25 Transformers Course (OPEN TO ALL | Starts Tomorrow) — Transformers are highly expressive, optimizable, and efficient general-purpose computing models for diverse AI tasks.
• An MLIR-Based Compilation Method for Large Language Models — MLIR-based compilation with a three-stage Transformer layer split optimizes LLM deployment on specialized AI accelerators.
• Explaining Attention with Program Synthesis — Program synthesis can create human-readable code to explain and replace transformer attention heads.
• My Workflow for Understanding LLM Architectures — Inspect open-weight LLM code and config files to understand architectures when papers lack detail.
• Scaling Interpretable Transformers with Parity Bottleneck Layers — ParityTransformer integrates efficient, sparse bottlenecks directly into Transformer architecture for scalable, interpretable models.
• How Andrej Karpathy Built a Working Transformer in 243 Lines of Code — microGPT offers a transparent, dependency-free Python implementation of GPT-2 fundamentals for educational purposes.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.