LLM profiling in CI/CD: evaluating inference, chip by chip

2026-07-19 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, short

Summary

This article details a hardware-agnostic model profiling harness designed to accurately evaluate LLM inference performance by dissecting the "tokens per second" metric. It highlights that this single figure obscures two distinct phases: prefill, which processes the prompt in one parallel pass and is compute-bound, and decode, which generates tokens autoregressively and is memory-bound due to repeated data movement. The profiling method, utilizing PyTorch and operator-level tracing on an open-weights Granite model, reveals that prompt length primarily impacts Time To First Token (TTFT) during prefill, while inter-token latency (p50 and p95) characterizes decode. Crucially, quantization, like INT8, can improve decode speed but potentially degrade long-prompt prefill performance and even increase peak memory. The analysis emphasizes that user experience is often dictated by p95 latency, not just p50.

Key takeaway

For MLOps Engineers optimizing LLM inference, relying solely on "tokens per second" is insufficient; you must profile prefill and decode phases independently. Instrumenting your models with tools like PyTorch's profiler to track Time To First Token (TTFT) and inter-token latency (p50/p95) will reveal true bottlenecks, especially regarding memory traffic during decode. This granular insight allows you to make informed hardware and quantization decisions, ensuring user-perceived performance (p95) is prioritized over misleading averages.

Key insights

LLM inference performance is split between compute-bound prefill and memory-bound decode, requiring distinct profiling.

Principles

• Prefill is compute-bound; decode is memory-bound.
• Quantization can improve decode but degrade long-prompt prefill.
• P95 latency reflects user experience more than p50.

Method

Run inference as two explicit phases (prefill/decode). Use PyTorch, operator-level tracing, and Perfetto for span duration queries. The method is hardware-agnostic.

In practice

• Instrument prefill and decode separately.
• Monitor TTFT for prefill, p50/p95 for decode.
• Evaluate quantization impact on both phases.

Topics

• LLM Inference
• Performance Profiling
• Prefill Decode
• Compute-Memory Bottlenecks
• Quantization (INT8)
• PyTorch Profiler

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• Inside LLMs #1 — The Anatomy of an LLM Response — LLM inference involves compute-bound prefill for TTFT and memory-bandwidth-bound, sequential decode for TPOT, each with distinct bottlenecks.
• Qwen3.5 27B Latency and Throughput: INT4 vs NVFP4 vs FP8 vs BF16 — Quantization improves LLM memory efficiency and single-query latency, but saturated workloads face GPU memory bandwidth limits.
• Your LLM Isn’t Slow Because of the Model. It’s Slow Because of Physics. — LLM inference speed is often limited by data movement (memory-bound) rather than raw computation (compute-bound).
• Your LLM Is Guessing Ahead. Then It Checks Itself aka Speculative Decoding — Speculative decoding accelerates LLM inference by using a small model to guess tokens, verified by a large model in one pass.
• Inference Optimization — How to Make LLMs Faster and Cheaper in Production — Optimizing LLM inference in production requires a stack of techniques to balance latency, throughput, and cost.
• AI 101: From Tokens to Answers: What Actually Happens During LLM Inference — Modern LLM inference is a complex system orchestration problem balancing latency, cost, and resource efficiency across distinct prefill and decode phases.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.