Accelerating Large Language Model Inference with Self-Supervised Early Exits

2024-07-30 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A new modular approach accelerates large language model (LLM) inference by integrating self-supervised early exit heads into intermediate transformer layers. These heads mimic the main model's predictions, allowing computation to halt early when a calibrated confidence threshold is met. Experiments on the Pythia model suite (70M to 2.8B parameters) show significant inference cost reduction, maintaining accuracy across benchmarks. Entropy proved the most reliable confidence metric. This method also adapts into Dynamic Self-Speculative Decoding (DSSD). DSSD achieves 1.66x higher token acceptance than LayerSkip baselines with minimal hyperparameter tuning.

Key takeaway

For AI Engineers optimizing LLM deployment costs, this self-supervised early exit method offers a direct path to significant inference acceleration. You can reduce computational overhead by integrating these heads into your transformer models, leveraging entropy for reliable early stopping. Consider implementing Dynamic Self-Speculative Decoding (DSSD). This can achieve superior token acceptance rates over traditional LayerSkip approaches, minimizing manual tuning efforts.

Key insights

Self-supervised early exits accelerate LLM inference by stopping computation early based on confidence, improving efficiency.

Principles

• Mimic main model predictions.
• Calibrate confidence thresholds.
• Entropy reliably gauges prediction.

Method

Add early exit heads to intermediate transformer layers. Train heads self-supervised to mimic main model. Stop inference when calibrated confidence (e.g., entropy) is met.

In practice

• Implement early exit heads.
• Use entropy for confidence.
• Apply to speculative decoding.

Topics

• Large Language Models
• LLM Inference Acceleration
• Early Exit Networks
• Self-Supervised Learning
• Speculative Decoding
• Pythia Models

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• The Evolution of LLM Inference: Decoding algorithms — Part 1 — LLM inference speed hinges on reducing sequential decoding steps by predicting and verifying multiple tokens in parallel.
• A team of researchers form Meta and Stanford Propose Fast Byte Latent Transformer That Reduces Inference Memory Bandwidth by Over 50% Without Tokenization — New BLT methods drastically cut byte-level model inference memory bandwidth without tokenization or quality loss.
• S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation — S2D2 accelerates block-diffusion LLMs by reusing the same model as both drafter and autoregressive verifier.
• ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding — ConfLayers uses confidence-based layer skipping to accelerate LLM inference without training a separate policy.
• Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction — Hyper-Parallel Decoding accelerates LLM inference by exploiting output independence for parallel token generation.
• Accelerating decode-heavy LLM inference with speculative decoding on AWS Trainium and vLLM — Speculative decoding accelerates LLM inference for predictable outputs by reducing sequential decode steps and improving hardware utilization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.