OpenAI Used Population-Level Core-Dump Analysis to Trace Crashes to a Bad Azure Host and an 18-Year-Old libunwind Bug
OpenAI engineers investigated mysterious crashes in their Rockset C++ data infrastructure by applying an epidemiological approach to a year’s worth of core dumps, uncovering two unrelated bugs: silent hardware corruption on a single Azure host and an 18-year-old race condition in GNU libunwind.