How I topped the Open LLM Leaderboard using 2x 4090 GPUs - Research notes in Blog form

2026-03-10 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Large Language Models · Depth: Advanced, medium

Summary

An independent researcher discovered that duplicating a specific block of approximately seven middle layers within the Qwen2-72B model significantly improved its performance across all Open LLM Leaderboard benchmarks, leading to a #1 ranking. This technique, developed on two RTX 4090 GPUs, suggests that pre-training carves out discrete, functional "circuits" within a Transformer's layer stack that operate robustly even when rearranged or duplicated. The finding implies that Transformer layers are more interchangeable than previously assumed, capable of processing out-of-order hidden states without collapse. This method offers a "free upgrade" for models by reusing layers in VRAM, potentially reducing the need for extensive fine-tuning or RLHF to enhance model intelligence.

Key takeaway

For AI Engineers and Research Scientists optimizing LLM performance on consumer hardware, you should explore layer duplication as a novel, low-compute method. This technique, demonstrated to improve Qwen2-72B's leaderboard standing, suggests that simply giving a model "more layers to think with" via architectural rearrangement can yield significant gains without retraining. Consider replicating this approach with your existing models to potentially achieve a "free upgrade" in performance and efficiency.

Key insights

Duplicating specific Transformer layer blocks can significantly boost LLM performance without weight modification.

Principles

• Transformers form robust, functional layer circuits.
• Internal representations are homogenous and flexible.
• Layer duplication can enhance model reasoning.

Method

Identify and duplicate a specific block of ~7 middle layers in a pre-trained Transformer model. This reuses layers in VRAM, requiring a new KV cache but no weight changes.

In practice

• Experiment with duplicating 7-layer blocks in Qwen-like models.
• Investigate looping circuits instead of outright duplication.
• Test this method on other Transformer architectures.

Topics

• LLM Architecture
• Layer Duplication
• Transformer Circuits
• Open LLM Leaderboard
• Low-Compute AI

Code references

• dnhkng/GLaDOS

Best for: AI Scientist, Research Scientist, AI Engineer, AI Researcher, Machine Learning Engineer, Deep Learning Engineer

Related on AIssential

• The Ultimate Transformer Course for Working Engineers — The course offers practical understanding of transformers to debug LLM issues like slow inference and OOM errors.
• The Capability Frontier: Benchmarks Miss 82% of Model Performance — LLM benchmarks significantly understate collective capabilities due to single-model, single-run evaluations, missing 82% of potential performance.
• Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance — Recent LLMs can use filler tokens or problem repeats to boost math performance without explicit Chain-of-Thought.
• The Big LLM Architecture Comparison — Modern LLM architectures prioritize inference efficiency and training stability through diverse attention mechanisms and Mixture of Experts.
• LLMs (Part-01): The High-level Architecture of Transformers — The Transformer architecture, detailed in "Attention is All You Need," forms the basis for modern LLMs through its encoder-decoder structure and attention mechanisms.
• Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization — Parallel Track Transformers reduce GPU synchronization for faster, more efficient LLM inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.