Core dump epidemiology: fixing an 18-year-old bug

2026-05-27 · Source: OpenAI News · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Expert, long

Summary

OpenAI's engineering team, on June 30, 2026, successfully debugged an 18-year-old race condition in GNU libunwind and a hardware corruption issue affecting their C++ data infrastructure, Rockset. Initially, inexplicable crashes in Rockset, a key component for ChatGPT's data plugins and conversation search, were observed, manifesting as returns to bogus addresses or misaligned stack pointers. The team shifted from individual core dump analysis to population-level epidemiology, building a pipeline to automatically analyze core dumps. This revealed two distinct crash populations: misaligned-stack crashes linked to a single faulty Azure host, and return-to-null crashes caused by a one-instruction race window in GNU libunwind's `_Ux86_64_setcontext` during C++ exception unwinding. The libunwind bug, present for 18 years, became operationally visible due to Rockset's high exception rate, frequent `SIGUSR2` signal delivery, and increased signal handler stack usage. OpenAI denylisted the bad host and switched to libgcc's unwinder, also upstreaming a fix to GNU libunwind.

Key takeaway

For MLOps Engineers debugging intermittent, hard-to-diagnose system crashes, shift from single-case analysis to population-level data epidemiology. Automate core dump analysis to identify distinct crash patterns and underlying causes, like hardware faults or subtle race conditions. This approach prevents conflating issues, enabling targeted fixes and improving overall system reliability. Update your operational tooling and runbooks to incorporate this data-driven debugging strategy.

Key insights

Population-level analysis of crash data can reveal distinct, otherwise inexplicable, underlying issues.

Principles

• Conflating distinct issues hinders debugging.
• High-quality population data is crucial.
• Operational visibility of old bugs depends on system load.

Method

Automate core dump analysis to extract registers, filter false positives, and label crash types, then analyze the resulting dataset for population-level correlations across infrastructure.

In practice

• Implement automated core dump analysis.
• Improve fatal signal handlers for detailed logging.
• Switch to robust, well-maintained libraries.

Topics

• Core Dump Analysis
• C++ Debugging
• Race Conditions
• GNU libunwind
• Azure Infrastructure
• Population-Level Analysis

Code references

• libunwind/libunwind

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Software Engineer, MLOps Engineer, AI Engineer

Related on AIssential

• datasette 1.0a29 — Race conditions in resource management can lead to severe, difficult-to-diagnose bugs like segfaults.
• Data Fusion for High-Resolution Estimation — Fusing biased high-resolution surveys with unbiased low-resolution aggregate data improves granular estimates by modeling sampling bias as exponential tilting.
• Code Lifespan Survival Analysis (CLSA): Predicting the Survival of Source Code Lines Using AST-Aware Mining — Predicting individual source code line deletion risk is feasible using static features and survival analysis, revealing time-varying effects.
• I Built a Data Quality Tool Because I Was Tired of Debugging Bad Data — Fast, opinionated first-contact profiling for tabular data prevents downstream ML issues.
• Demystifying Residual Analysis: A Beginner’s Guide to What Your Model Isn’t Telling You — Residual analysis diagnoses model failures by examining prediction errors for hidden patterns and assumption violations.
• Do a sanity check on your experiments — Manual sanity checks on data, model outputs, and evaluations are crucial for detecting common experimental bugs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by OpenAI News.