Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates

2026-02-25 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Speech and Natural Language Processing · Depth: Advanced, quick

Summary

A novel method called Constructive Circuit Amplification (CCA) has been developed to improve mathematical reasoning in Large Language Models (LLMs) by targeting specific sub-networks. This approach identifies pivotal tokens in reasoning traces and the model components responsible for a desired task, then updates only those components. Applied to mathematical reasoning, CCA enhances accuracy by up to +11.4% across multiple models while modifying as little as 1.59% of model components. Crucially, this targeted intervention has minimal impact on other abilities, as measured by benchmarks like MMLU, TriviaQA, and TruthfulQA, demonstrating that specific capabilities can be reliably improved through sparse, selective updates.

Key takeaway

For research scientists focused on fine-tuning LLMs for specialized tasks, you should consider adopting methods like Constructive Circuit Amplification. This approach allows for significant performance gains in specific domains, such as mathematical reasoning, by making highly targeted updates to a minimal fraction of the model, thereby preserving broader capabilities and reducing the risk of unintended regressions.

Key insights

Targeted sub-network updates can significantly improve specific LLM capabilities with minimal collateral impact.

Principles

• LLM performance gains often strengthen existing circuits.
• Sparse subnetworks perform specific tasks.
• Targeted updates enhance specific capabilities.

Method

Constructive Circuit Amplification identifies pivotal tokens and task-specific model components, then updates only those selected components to enhance performance.

In practice

• Improve math reasoning in LLMs by +11.4%.
• Modify <2% of model components.
• Maintain performance on other benchmarks.

Topics

• Constructive Circuit Amplification
• LLM Mathematical Reasoning
• Sparse Sub-network Updates
• Model Interpretability
• Targeted Fine-tuning

Best for: Research Scientist, AI Researcher, AI Scientist, Deep Learning Engineer

Related on AIssential

• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning — Larger LLMs demonstrate superior efficiency and performance in RL post-training for mathematical reasoning.
• Selective Neuron Amplification for Training-Free Task Enhancement — Weak neuron activation, not missing knowledge, often causes LLM failures, addressable via inference-time amplification.
• Granularity-Regulated Adaptive Computational Efficiency for Optimal Verification in Test-Time Scaling — Optimal LLM verification granularity depends on problem difficulty and compute budget, exhibiting a phase transition.
• The Logic Auditor: Why Your LLM Needs a “Constructive Lie” to Achieve 99% Accuracy — Adversarial reasoning, using "Constructive Lies," forces LLMs into verification mode, significantly improving accuracy and grounding.
• Demystifying Variance in Circuit Discovery of LLMs — LLM circuit discovery faces significant variance; CEAP reduces resampling, but rephrasing variance implies inherent LLM steering difficulty.
• TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks — Targeted, step-level LLM interventions significantly boost inference efficiency and accuracy in multi-step reasoning tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.