Inside LLMs #1 — The Anatomy of an LLM Response

2026-07-26 · Source: Artificial Intelligence on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

This article details the internal process of Large Language Model (LLM) inference, dissecting the journey from prompt submission to response generation. It introduces two key metrics: Time To First Token (TTFT), covering initial processing, and Time Per Output Token (TPOT), measuring subsequent token generation speed. The process begins with network and authentication overhead, typically 80-150 ms, followed by queuing. The core inference involves "prefill," a compute-bound stage where the entire prompt is processed in parallel to create a KV cache, directly impacting TTFT. Subsequently, "decode" is a memory-bandwidth-bound stage where tokens are generated sequentially, one at a time, relying on the KV cache. GPUs excel at prefill's parallel compute but struggle with decode's memory-intensive, sequential nature. The article notes that network RTT, security checks, and decode's inherent sequentiality are fundamental, unoptimizable limitations.

Key takeaway

For AI Engineers optimizing LLM inference, understanding the distinct prefill and decode stages is crucial. You should focus on compute optimization for Time To First Token (TTFT) during prefill, and memory bandwidth optimization for Time Per Output Token (TPOT) during decode. Recognize that network latency and decode's sequential nature are fundamental limits, guiding your efforts towards areas like KV cache and FlashAttention for tangible gains.

Key insights

LLM inference involves compute-bound prefill for TTFT and memory-bandwidth-bound, sequential decode for TPOT, each with distinct bottlenecks.

Principles

• Prefill is compute-bound, maximizing parallel throughput.
• Decode is memory-bandwidth-bound and sequential.
• TTFT equals queue time plus prefill time.

Topics

• LLM Inference
• Time To First Token
• Time Per Output Token
• Prefill Stage
• Decode Stage
• KV Cache
• GPU Bottlenecks

Best for: Machine Learning Engineer, AI Engineer, AI Architect

Related on AIssential

• AI 101: From Tokens to Answers: What Actually Happens During LLM Inference — Modern LLM inference is a complex system orchestration problem balancing latency, cost, and resource efficiency across distinct prefill and decode phases.
• LLM Inference and Optimization: Fundamentals, Bottlenecks, and Techniques — Optimizing LLM inference requires understanding distinct prefill and decode phases, leveraging KV caching, and applying advanced techniques.
• LLM profiling in CI/CD: evaluating inference, chip by chip — LLM inference performance is split between compute-bound prefill and memory-bound decode, requiring distinct profiling.
• Your LLM Is Guessing Ahead. Then It Checks Itself aka Speculative Decoding — Speculative decoding accelerates LLM inference by using a small model to guess tokens, verified by a large model in one pass.
• Prefill/Decode Disaggregation: Why Your GPU Can’t Do Two Things at Once — LLM inference's prefill and decode phases have conflicting hardware demands, necessitating disaggregation for optimal performance and user experience.
• What Actually Happens When You Send a Prompt… — The LLM prompt-to-response journey involves encryption, tokenization, embedding, attention, and two-phase inference (prefill/decode) before output.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence on Medium.