TraceLens: Democratizing AI Performance Analysis

2026-04-27 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

TraceLens, an open-source tool released on April 27, 2026, by Adeem Jassani et al. at AMD, democratizes AI performance analysis by converting complex framework profiler traces into structured summaries and comparisons. It supports PyTorch and JAX, working with backends like ROCm and CUDA. Key capabilities include Trace2Tree, which builds a hierarchical event tree linking Python operations to GPU kernels, and Hierarchical Performance Breakdowns for identifying bottlenecks. TraceLens also offers Compute & Roofline Modeling to assess kernel efficiency (TFLOPS/s, TB/s), Multi-GPU Communication Analysis to diagnose scaling issues by separating communication time from synchronization skew, and Trace Comparison for quantifying the impact of changes. Additionally, its Event Replay feature generates minimal, self-contained scripts for debugging specific operations in isolation, and an extensible Python SDK allows for custom analyses.

Key takeaway

For NLP or Computer Vision Engineers optimizing AI workloads, TraceLens offers a powerful, open-source solution to quickly diagnose performance bottlenecks. You should integrate TraceLens into your profiling workflow to transform raw traces into actionable insights, enabling you to pinpoint slow kernels, assess multi-GPU scaling, and quantify the impact of code or hardware changes more efficiently. This will accelerate your debugging and optimization cycles.

Key insights

TraceLens transforms complex AI profiler traces into actionable insights for performance optimization across various backends.

Principles

• Hierarchical analysis simplifies bottleneck identification.
• Isolate true communication from synchronization skew.
• Quantify efficiency with theoretical compute models.

Method

TraceLens consumes framework traces, converts them into a hierarchical event tree (Trace2Tree), then applies various analysis modules for performance breakdowns, compute modeling, multi-GPU communication, and trace comparison.

In practice

• Use Trace2Tree to link Python ops to GPU kernels.
• Generate minimal reproducers for specific operations.
• Compare traces to quantify performance changes.

Topics

• TraceLens
• AI Workload Optimization
• Hierarchical Trace Analysis
• Multi-GPU Scaling
• Roofline Modeling

Code references

• AMD-AGI/TraceLens
• ROCm/rocm-systems
• AMD-AGI/TraceLens

Best for: NLP Engineer, Computer Vision Engineer, Machine Learning Engineer, MLOps Engineer, AI Engineer

Related on AIssential

• Chakra Comes of Age: A Standardized Trace Ecosystem for AI Systems Benchmarking and Co-design — Standardized execution traces enable open, interoperable AI system co-design and performance benchmarking across the stack.
• [P] Zero-code runtime visibility for PyTorch training — TraceML's zero-code mode offers live PyTorch training visibility via `traceml watch train.py`.
• Finding zombies in our systems: A real-world story of CPU bottlenecks — Intermittent network issues on Ray ML jobs stemmed from CPU starvation caused by zombie memory cgroups from a crashing ECS agent.
• Arm launches Performix toolkit for AI workload optimisation — Arm's Performix toolkit automates AI workload optimization by integrating performance analysis into modern development workflows.
• Real-Time Performance Monitoring and Faster Debugging with NCCL Inspector and Prometheus — NCCL Inspector's Prometheus Mode enables real-time GPU communication monitoring for AI workloads, improving performance debugging.
• Top 10: AI Benchmarking Tools — AI benchmarking is critical for validating performance, safety, and compliance across increasingly complex models and hardware.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.