Super Weights in LLMs and the Failure of Selective Training

2026-07-09 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Recent research challenges the universal importance and trainability of "Super Weights" in Large Language Models (LLMs), parameters previously identified as critical for model performance. A study using OLMo-1B and OLMo-7B models demonstrates that pruning Super Weights does not consistently degrade all LLMs. Crucially, attempts to train Super Weights in isolation, ranging from 100 to 8,192 parameters, or even within local neighborhoods up to 36K parameters, resulted in accuracy levels equivalent to random guessing. This performance collapse is specific to Super Weight coordinates, as training an equal number of randomly chosen parameters in the same down_proj layers actually improved over the baseline. In contrast, vanilla LoRA, which updates 0.16% of parameters through low-rank structured decompositions across entire layers, proved effective, even when applied to down_proj layers. These findings indicate that while certain parameters may be important, they are not necessarily trainable in isolation, suggesting that effective fine-tuning strategies should focus on structured, layer-wide updates rather than targeting individual "Super Weights."

Key takeaway

For Machine Learning Engineers optimizing LLM fine-tuning, reconsider strategies focused on individually "important" parameters. Your efforts to selectively train or prune specific "Super Weights" are unlikely to yield performance gains and may degrade models like OLMo-1B and OLMo-7B. Instead, prioritize structured, low-rank adaptation methods such as LoRA, which effectively update entire layers with minimal parameters (e.g., 0.16%). This approach offers superior results compared to targeting isolated weights, even those deemed critical.

Key insights

Parameter importance does not imply trainability; effective LLM fine-tuning requires structured, layer-wide updates.

Principles

• Pruning Super Weights does not universally degrade all LLMs.
• Training Super Weights in isolation fails, even locally.
• Effective fine-tuning uses structured decompositions over layers.

In practice

• Avoid isolated training of "important" parameters.
• Prioritize low-rank structured updates like LoRA.
• Focus fine-tuning on entire layers, not specific weights.

Topics

• Super Weights
• LLM Fine-tuning
• Parameter Pruning
• LoRA
• OLMo Models
• Low-Rank Adaptation

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning — Pruning-inspired saliency signals can effectively guide sparse PEFT for LLMs, especially when combined with low-rank adapters.
• LoRA, Explained With Spreadsheets: How to Train a Model Without Touching Its Weights — LoRA enables efficient model training by adapting a small, low-rank matrix instead of modifying all original weights.
• Build an LLM from Scratch 6: Finetuning for Classification — Fine-tuning pre-trained LLMs for specific classification tasks is efficient and effective, even with smaller models.
• Optimizing LoRA target module selection for efficient fine tuning — Strategic LoRA target module selection significantly improves LLM fine-tuning efficiency and accuracy.
• EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning — Aligning parameter-efficient fine-tuning with MoE routing dynamics significantly improves adaptation efficiency and performance.
• Understanding post-training of LLMs: SFT — Post-training guides LLM behavior and expression, not knowledge, primarily via SFT and preference optimization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.