學習筆記｜機器學習系統架構 AMLS — 02 Compilation — Size Inference and Rewrites

2026-07-12 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, quick

Summary

The article details how machine learning systems optimize execution plans during the compilation phase through Size Inference and Rewrites. It outlines four key areas: Compilation Overview, Size Inference and Cost Estimation, Rewrites, and Physical Optimization & Operator Selection. Compilation strategies range from interpretation to full program compilation, often utilizing Directed Acyclic Graphs (DAGs) to represent code for global optimization. Size inference estimates data dimensions and sparsity, propagating information bottom-up through DAGs and across control flow structures like "if"/"loops" and functions via Intra/Inter-Procedural Analysis, often using worst-case estimates. Rewrites involve traditional compiler optimizations (e.g., Common Subexpression Elimination, Constant Folding) and ML-specific static and dynamic transformations based on mathematical theorems or inferred data metadata. Finally, physical optimization addresses distributed environment bottlenecks through distributed caching, data partitioning, and data flow optimization, culminating in the selection of appropriate local, broadcast-based, or shuffle-based operators.

Key takeaway

For Machine Learning Engineers optimizing model deployment, understanding compilation techniques like size inference and rewrites is crucial. This knowledge enables you to diagnose performance bottlenecks, anticipate memory requirements, and select appropriate operators for efficient execution. By leveraging these compiler optimizations, you can significantly reduce computational costs and improve the scalability of your ML systems.

Key insights

ML compilation optimizes execution plans through size inference and logical rewrites, enhancing efficiency before runtime.

Principles

• Compilation optimizes ML execution before runtime.
• Size inference and rewrites are core to plan optimization.

Method

ML compilation involves size inference via DAG, program-level, and IPA; logical rewrites (CSE, constant folding, simplification); then physical optimization and operator selection.

In practice

• Utilize DAGs for compiler code representation.
• Apply Common Subexpression Elimination (CSE).
• Implement data partitioning for distributed ML.

Topics

• Machine Learning Systems
• Compiler Optimization
• Size Inference
• Code Rewriting
• Directed Acyclic Graph
• Operator Selection

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• When AI Learns to Tune Itself: How ML Is Rewriting the Rules of Compiler Optimization — Published on June 11th, 2026, by Ankit Sinha, a Tech Lead at Google specializing in ML/AI infrastructure, this article examines the profound impact of machine learning on compiler optimization.
• Train-to-Test scaling explained: How to optimize your end-to-end AI compute budget for inference — Train-to-Test (T2) scaling laws optimize LLM compute across training and inference, favoring smaller, overtrained models for reasoning tasks.
• An MLIR-Based Compilation Method for Large Language Models — MLIR-based compilation with a three-stage Transformer layer split optimizes LLM deployment on specialized AI accelerators.
• Rewriting model inference with CUDA kernels: the bottleneck was not just GEMM [P] — Small-batch ML inference bottlenecks often stem from runtime glue, not just GEMM, requiring direct kernel optimization.
• Token-Operations-Oriented Inference Optimization Techniques for Large Models — Token-oriented inference optimization requires a four-layer architecture to balance quality, cost, and stability for large model services.
• CUSUM-Shaped Inference-Time Monitoring and Targeted Re-Decoding for Quantized Small Language Model Reasoning — MGT-B selectively monitors and repairs unproductive reasoning trajectories in quantized small language models during inference.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.