Cross-Trajectory Chimera Interventions Reveal Dissociable Roles of Weight Magnitude and Direction in Grokking

2026-07-07 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new research method, cross-trajectory chimera interventions, reveals dissociable roles for weight magnitude (norm) and direction in neural network grokking. This technique involves splitting weight vectors from two independently trained networks into their norm and unit direction, then recombining one run's norm with another's direction for continued training. Applied to modular-arithmetic tasks exhibiting grokking, the study found that weight direction carries a transferable, donor-specific circuit identity. Implanting a donor's direction into a recipient's norm drove the network to the donor's circuit in 40/40 cases, unlike random controls. This transfer is threshold-like, predicted by the recipient's norm, separating perfectly by norm class over all 20 pairs (joint permutation probability 1.9e-4). Conversely, weight norm primarily contributes a modest, distributed delay effect and influences how susceptible a circuit identity is to being overwritten, with the threshold localized to +/-1/64.

Key takeaway

For research scientists investigating neural network learning and generalization, understanding that weight direction encodes circuit identity is crucial. Your focus should shift beyond overall weight values to the specific roles of direction and magnitude. This insight suggests new avenues for steering network behavior or transferring learned capabilities by manipulating these distinct components, particularly when dealing with grokking phenomena. Consider designing experiments that selectively modify weight directions to implant specific circuit identities.

Key insights

Weight direction dictates a network's learned circuit identity, while magnitude influences its susceptibility to change during grokking.

Principles

• Network weight direction encodes circuit identity.
• Weight norm governs identity susceptibility.
• Causal portability requires cross-trajectory analysis.

Method

Cross-trajectory chimera interventions split weight vectors into norm and direction, recombining components from different training runs, then continue training to observe causal portability. An adaptive bisection procedure localizes thresholds.

In practice

• Analyze weight components for circuit identity.
• Use chimera interventions for causal portability.
• Investigate norm thresholds for identity stability.

Topics

• Grokking
• Neural Network Weights
• Weight Direction
• Weight Magnitude
• Modular Arithmetic
• Causal Interventions

Best for: AI Scientist, Research Scientist

Related on AIssential

• What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy — Weight norm controls grokking delay primarily through logit scale and softmax saturation, especially under cross-entropy.
• From Data Statistics to Feature Geometry: How Correlations Shape Superposition — Feature correlations can enable constructive interference in neural network superposition, forming semantic clusters.
• Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics — Gradient descent in ReLU networks can be modeled as collective dynamics in training-set space, mediated by layer-wise Gram operators.
• Circuit Synchronization Precedes Generalization: Causal Evidence from Fourier Structure in Grokking Transformers — Frequency Synchronization Degree (FSD) causally predicts grokking onset by tracking Fourier circuit synchronization, linking it to regularization.
• Memory-Augmented Spiking Networks: Synergistic Integration of Complementary Mechanisms for Neuromorphic Vision — Architectural balance of memory mechanisms in SNNs yields synergistic performance gains over individual optimizations.
• Why Every Weight in a Neural Network Is Born Divided by the Square Root of n. — Proper neural network weight initialization prevents exploding or vanishing signals, ensuring stable learning from the start.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.