Exercises in benchmarking and evals, part 7: DeepSWE, Senior SWE-Bench, napkin math, and winter tires

2026-07-22 · Source: danluu.com - danluu.com · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Advanced, extended

Summary

This analysis critically examines three distinct benchmarking scenarios, revealing common pitfalls in performance measurement and evaluation. The popular "sirupsen/napkin-math" GitHub repository, providing computer performance estimates, contains inaccuracies, including an understated 20ns random memory R/W latency (expected ~100ns) and flawed SSD read benchmarks that inadvertently read from cache or use unaligned accesses. Its sequential SSD read claim of 8 GiB/s exceeds documented cloud instance limits. AI coding agent benchmarks like DeepSWE and Senior SWE-Bench are also scrutinized for presenting implausible results, such as GPT-5.5 outperforming Opus 4.8, which contradicts user experience. These AI benchmarks employ subjective grading criteria, arbitrary length cutoffs for "tasteful solves," and exhibit significant LLM grading variance. Finally, the widespread belief that all-season tires become too hard below 7°C (45°F) is challenged by empirical tire performance tests, which demonstrate that all-season tires often outperform winter tires in dry or wet conditions above 0°C.

Key takeaway

For software engineers and AI scientists evaluating system performance or model capabilities, you should critically examine benchmark methodologies. Do not rely solely on headline scores from popular "napkin math" repositories or AI agent leaderboards like DeepSWE and Senior SWE-Bench. Instead, scrutinize the underlying code for caching issues, assess task representativeness, and be wary of subjective, discontinuous scoring. Your decisions on system design or model selection should be informed by benchmarks that reflect real-world workloads and employ robust, continuous evaluation metrics.

Key insights

Flawed methodology, unrepresentative tasks, and subjective scoring undermine benchmark validity and generalizability.

Principles

• Benchmarks must prevent unintended caching effects.
• Single summary scores often obscure true performance.
• Discontinuous scoring introduces arbitrary thresholds.

Method

To accurately measure memory latency, introduce data dependencies between loads to prevent parallel execution. For disk performance, account for read size, queue depth, and job count.

In practice

• Scrutinize benchmark code for caching issues.
• Validate LLM benchmark tasks against real workloads.
• Avoid benchmarks with arbitrary pass/fail cutoffs.

Topics

• Benchmarking
• Performance Evaluation
• AI Coding Agents
• LLM Evals
• System Performance
• Experimental Design

Code references

• sirupsen/napkin-math

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• The Sequence AI of the Week #895: OpenAI's Show Us Where Coding Evals Break — Precision in coding benchmarks does not guarantee validity; flawed evaluations can misrepresent model capabilities.
• Finally a good benchmark (DeepSWE) — DeepSWE offers a robust, real-world software engineering benchmark revealing significant performance and cost disparities among LLMs.
• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering — Current coding benchmarks fail to accurately evaluate agentic software engineering due to conflation, single-reference grading, and lack of component-level signal.
• SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks — Dynamic, continuously updated benchmarks are crucial for reliable LLM evaluation in software engineering, mitigating data contamination and saturation.
• SWE-Bench is getting replaced??? — Existing AI coding benchmarks are flawed by contamination and unrealistic tasks, while DBSE offers a more accurate, behavior-focused evaluation.
• [D] What is even the point of these LLM benchmarking papers? — LLM benchmarking papers often become obsolete quickly due to rapid model updates, but their datasets can still be valuable.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by danluu.com - danluu.com.