Why Is My Code So Slow? A Guide to Py-Spy Python Profiling

2026-02-05 · Source: Towards Data Science · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, medium

Summary

The `py-spy` tool is introduced as a powerful sampling profiler for identifying and correcting inefficiencies in Python data science code. The article demonstrates `py-spy`'s utility by optimizing a script that calculates Haversine distances for 3.5 million flight records from January to June 2025 BTS data, initially taking 169.89 seconds. `py-spy` generates an interactive Icicle Graph, which revealed that the Pandas `iterrows()` function consumed 68.36% of the runtime. By replacing the `iterrows()` loop with a vectorized NumPy-based Haversine calculation, the script's execution time was reduced to 0.56 seconds, achieving identical results. This significant performance improvement highlights `py-spy`'s effectiveness in diagnosing and resolving bottlenecks in Python programs.

Key takeaway

For Data Scientists and Software Engineers struggling with slow Python scripts, `py-spy` offers a critical diagnostic tool. Use it to generate an Icicle Graph, identify performance bottlenecks like `iterrows()`, and then refactor your code with vectorized operations (e.g., NumPy) to achieve substantial speedups. This approach can transform a multi-minute wait into sub-second execution, significantly improving development workflow and pipeline efficiency.

Key insights

`py-spy` is a sampling profiler that efficiently pinpoints Python code bottlenecks without significant overhead.

Principles

• Sampling profilers offer low overhead.
• Vectorized operations are faster than row-wise loops.

Method

`py-spy` records program state snapshots at high frequency, generating an Icicle Graph to visualize time spent in functions, with wider bars indicating more frequent execution.

In practice

• Use `py-spy record -o profile.svg -r 100 -- python main.py`.
• Analyze Icicle Graphs: wider bars mean more time.
• Replace `iterrows()` with vectorized NumPy operations.

Topics

• Python Profiling
• py-spy
• Performance Optimization
• Data Science Workflows
• Vectorization

Code references

• KenMcCarthy24/language_substring_analysis

Best for: Data Scientist, Data Engineer, Software Engineer

Related on AIssential

• 3 NumPy Tricks for Numerical Performance — Optimize NumPy performance by utilizing vectorization, in-place operations, and memory views to avoid Python overhead and unnecessary memory allocations.
• Clean your Python Pandas Code in Under 10 Minutes! — Chain Pandas commands and use `PyJanitor` to streamline data cleaning and improve code readability.
• I Compared 4 Python HTTP Libraries. One Shocked Me Completely. — The author, after experiencing significant performance bottlenecks with the `requests` library in a high-throughput system making hundreds of API calls per minute, undertook a comprehensive benchmark…
• Why Great Data Scientists Read More Python Code Than They Write (And How to Master It) — Efficient code reading demands a structural, architectural approach, not passive line-by-line parsing.
• I Reduced My Pandas Runtime by 95% — Here’s What I Was Doing Wrong — Efficient Pandas code prioritizes vectorized operations and memory management over row-wise processing for significant speed gains.
• How to Call Rust from Python — Integrating Rust with Python via Maturin and PyO3 can significantly accelerate performance-critical, non-vectorizable code sections.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.