Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

The Adaptive Depth Sparse Framework (AdaDSF) addresses the high inference costs of Transformer-based large language models (LLMs) by converting off-the-shelf pre-trained models into depth-sparse versions without requiring full retraining or task-specific fine-tuning. AdaDSF operates on the insight that LLM layers contribute unequally to representation transformation, quantifiable by the cosine similarity between layer input and output hidden states. It assigns layer-wise token retention ratios based on these similarity statistics, employs a lightweight router for informative token selection at each layer, and uses a feature-preserving alignment objective to match representations between sparse and dense models. Evaluated on GPT-NeoX and Qwen2.5 across language modeling and commonsense reasoning tasks, AdaDSF significantly reduces inference FLOPs while maintaining performance close to dense counterparts. It also demonstrates smaller accuracy degradation compared to baselines like MoD, D-LLM, and DLO under similar sparsity levels.

Key takeaway

For Machine Learning Engineers optimizing LLM deployment costs, you should consider the Adaptive Depth Sparse Framework (AdaDSF) to significantly reduce inference FLOPs for pre-trained models like GPT-NeoX and Qwen2.5. This method allows you to convert off-the-shelf LLMs into depth-sparse versions without full retraining, directly impacting operational expenses and resource allocation. Evaluate AdaDSF's applicability for your specific model architectures and tasks to achieve substantial efficiency gains while preserving performance.

Key insights

AdaDSF enables efficient, retraining-free depth-sparse conversion of LLMs by exploiting unequal layer contributions.

Principles

• LLM layers contribute unequally to representation transformation.
• Cosine similarity quantifies layer input/output hidden state transformation.

Method

AdaDSF assigns layer-wise token retention ratios from similarity statistics, uses a lightweight router, and applies a feature-preserving alignment objective.

In practice

• Convert off-the-shelf LLMs to depth-sparse models.
• Reduce inference FLOPs for GPT-NeoX and Qwen2.5.

Topics

• Large Language Models
• Model Sparsity
• Inference Optimization
• Transformer Architecture
• GPT-NeoX
• Qwen2.5

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs — Layer-wise compute allocation based on hidden-state similarity enables efficient depth-sparse LLMs without full retraining.
• DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression — Token-adaptive cross-layer residual factorization compresses KV caches, improving long-context LLM inference efficiency and quality without retraining.
• AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters — AdaFlash mitigates high variance in diffusion drafters via on-policy distillation and adaptive length adjustment for accelerated LLM inference.
• SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention — SparseBalance co-optimizes sparse attention and sequence length heterogeneity for efficient LLM training.
• Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priors — RAD3D-Prefix enables efficient 3D CT report generation by adapting frozen LLMs with diagnostic priors.
• RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention — RIS-Kernel enables long-context LLM inference on commodity CPUs by reducing self-attention complexity via sparse stochastic geometry.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.