Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

The Adaptive Depth Sparse Framework (AdaDSF) converts pre-trained large language models (LLMs) into efficient depth-sparse variants without full retraining or significant architectural changes. AdaDSF operates on the principle that Transformer layers contribute unequally to representation transformation, quantified by cosine similarity between layer input and output hidden states. It assigns layer-wise token retention ratios from these similarity statistics, employs a lightweight MLP-based router for dynamic token selection, and uses a feature-preserving alignment objective to match intermediate and final representations with the dense model. Evaluated on GPT-NeoX-130M, Qwen2.5-0.5B, and Qwen2.5-1.5B, AdaDSF substantially reduces inference FLOPs while maintaining performance close to dense baselines. For example, on GPT-NeoX-130M at 80% token retention, AdaDSF achieved a PPL of 18.9, outperforming MoD (21.6), D-LLM (2019), and DLO (19.6), with only 0.787x the FLOPs. This framework offers a 0.8%–4.7% average performance improvement, providing a practical efficiency-accuracy trade-off for LLM deployment.

Key takeaway

For MLOps Engineers deploying pre-trained LLMs in resource-constrained environments, you should consider AdaDSF to significantly reduce inference costs. This framework allows you to convert existing models like GPT-NeoX or Qwen2.5 into efficient sparse variants without extensive retraining. By dynamically allocating compute based on layer transformation, you can achieve better accuracy-efficiency trade-offs, preserving performance and reducing FLOPs by up to 21.3%. Evaluate AdaDSF for your specific models to optimize deployment.

Key insights

Layer-wise compute allocation based on hidden-state similarity enables efficient depth-sparse LLMs without full retraining.

Principles

• Layers transform representations unequally.
• Hidden-state similarity guides compute allocation.
• Aligning intermediate features preserves dense behavior.

Method

AdaDSF computes layer-wise token retention ratios from hidden-state cosine similarity, then uses a lightweight MLP router to select tokens. It trains with a feature-preserving alignment objective, matching intermediate and final representations between sparse and dense models.

In practice

• Convert pre-trained LLMs to sparse variants.
• Reduce inference FLOPs for deployment.
• Improve accuracy-efficiency trade-offs.

Topics

• Large Language Models
• LLM Inference Optimization
• Depth Sparsity
• Token Routing
• Hidden-State Similarity
• Feature Alignment

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs — AdaDSF enables efficient, retraining-free depth-sparse conversion of LLMs by exploiting unequal layer contributions.
• Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs — Jointly adapting visual tokens and LLM compute dynamically improves MLLM efficiency under varying content and budgets.
• When Data Is Scarce: Scaling Sparse Language Models with Repeated Training — Sparse training improves LLM scaling trade-offs and delays data saturation in data-scarce environments.
• DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression — Token-adaptive cross-layer residual factorization compresses KV caches, improving long-context LLM inference efficiency and quality without retraining.
• RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention — RIS-Kernel enables long-context LLM inference on commodity CPUs by reducing self-attention complexity via sparse stochastic geometry.
• Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space — DLCMs improve LLM efficiency by shifting computation from tokens to a compressed, concept-based reasoning space.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.