OpenAI Used Population-Level Core-Dump Analysis to Trace Crashes to a Bad Azure Host and an 18-Year-Old libunwind Bug
OpenAI engineers investigated mysterious crashes in their Rockset C++ data infrastructure by applying an epidemiological approach to a year’s worth of core dumps, uncovering two unrelated bugs: silent hardware corruption on a single Azure host and an 18-year-old race condition in GNU libunwind.
OpenAI described debugging anomalous crashes in Rockset, the C++ data infrastructure powering ChatGPT plugins and conversation search. The team observed two strange failure modes—functions returning to NULL addresses and misaligned stack pointers—that could not be explained through conventional, individual core-dump analysis. After building an automated pipeline (with scripting help from ChatGPT) to label and analyze a year of production crashes at population scale, correlations separated the incidents into two distinct causes: misaligned-stack crashes concentrated in one Azure region and traceable to a single faulty physical host with silent CPU corruption, and widespread return-to-null crashes attributed to an 18-year-old race condition in the open-source GNU libunwind library. Source: OpenAI Blog
The Wire · Newsletter
One careful email,
every Monday.
The week's most important AI stories, lightly edited and personally vouched for. No autoplay, no spam, easy to leave.
Comments · 0
Sign in to join the discussion.
Be the first to leave a thought.