Researchers Launch First Benchmark for Debugging LLM Multi-Agent Systems

When a multi-agent LLM system botches a task, figuring out which agent caused the failure is a nightmare. Researchers from Penn State University and Duke University, working with Google DeepMind and others, have formalized this problem and built the first benchmark to solve it.

The new research, spotlighted at ICML 2025, introduces "Automated Failure Attribution" — the challenge of pinpointing exactly which agent in a collaborative LLM chain is responsible for a breakdown, and when things went wrong. Multi-agent systems are proliferating across AI development, but diagnosing failures in them still means manually combing through sprawling interaction logs. It's slow, expensive, and scales poorly.

The team's benchmark dataset, called Who &When, gives the research community a standardized way to evaluate automated attribution methods for the first time. They also developed and tested several baseline approaches against it. Co-first authors Shaokun Zhang of Penn State and Ming Yin of Duke led the effort.

The code and dataset are fully open-source, so builders working on agent orchestration frameworks can start stress-testing their own failure-diagnosis pipelines now. For anyone shipping multi-agent products, faster root-cause analysis means faster iteration — and fewer hours buried in logs.