Friends Don’t Let Friends Run Loops Sequentially

2026-04-06 · Source: Microsoft Foundry Blog articles · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

Running Large Language Model (LLM) inference sequentially on datasets, even modest ones like 3,500 rows, can be extremely slow due to I/O-bound bottlenecks such as network latency and API response times. Microsoft data scientists encountered this issue when using an o3-mini LLM for text classification, where each inference call took approximately 2.5 seconds, leading to an estimated 146 minutes for the full dataset. To address this, they implemented parallel processing using Python's ThreadPoolExecutor, which allows multiple requests to be issued concurrently. This approach dramatically reduced runtime, achieving a 20x speedup on a 50-row sample, cutting processing time from 126.4 seconds to 6.2 seconds. Key considerations include checkpointing progress for resilience, tuning the number of workers to optimize against Azure OpenAI rate limits (Tokens Per Minute/Requests Per Minute), and understanding that parallelism saves time but not token-based billing costs.

Key takeaway

For Data Scientists and Machine Learning Engineers performing LLM-based data enrichment or classification on datasets, adopting parallel processing is crucial to drastically reduce execution time. You should implement concurrent API calls, ideally with checkpointing for resilience, and carefully tune your worker count to maximize throughput without exceeding provider rate limits. Remember that this optimization saves valuable time, allowing for faster iteration and analysis, but does not alter the token-based billing costs.

Key insights

Parallel processing significantly accelerates I/O-bound LLM inference by issuing concurrent requests, reducing wall-clock time.

Principles

• LLM inference is typically I/O-bound.
• Concurrency reduces wall-clock time, not token cost.
• Tune workers to avoid API rate limits.

Method

Implement parallel processing for LLM inference using a thread pool (e.g., Python's ThreadPoolExecutor) or async requests, incorporating checkpointing for fault tolerance and tuning worker count to optimize against API rate limits.

In practice

• Use ThreadPoolExecutor for I/O-bound LLM calls.
• Implement checkpointing to save progress.
• Experiment with worker count to maximize throughput.

Topics

• LLM Inference Optimization
• Parallel Processing
• I/O-bound Workloads
• Azure OpenAI Rate Limits
• ThreadPoolExecutor

Best for: Data Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient — Continuous batching optimizes LLM inference by dynamically scheduling requests and packing tokens to eliminate padding and idle GPU time.
• The Inference Reckoning: How to Stop Burning Millions on Cloud LLM Tokens — High-volume LLM inference demands dedicated infrastructure to avoid escalating cloud token costs and gain control.
• Streaming Faster Made Our LLM Hub Slower — Batching LLM tokens at the agent level significantly reduces network overhead and improves system stability under load.
• Your LLM Is Guessing Ahead. Then It Checks Itself aka Speculative Decoding — Speculative decoding accelerates LLM inference by using a small model to guess tokens, verified by a large model in one pass.
• How To Reduce Inference Costs While Running LLMs — Optimizing LLM inference costs and performance requires a multi-faceted approach across hardware, software, and architectural techniques.
• MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling — MARS reduces LLM test-time costs by adaptively stopping parallel reasoning traces while preserving accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Microsoft Foundry Blog articles.