Kuaishou's SRPO Cuts LLM Reasoning Training to a Tenth the Steps, Matches DeepSeek-R1-Zero

A team at Kuaishou called Kwaipilot has released SRPO, a reinforcement learning framework that matches DeepSeek-R1-Zero's reasoning performance on both math and code—using the same Qwen2.5-32B base model but only one-tenth the training steps. The researchers open-sourced the SRPO-Qwen-32B model and published a technical report detailing their method.

SRPO stands for Two-Staged history-Resampling Policy Optimization. It tackles two well-known pain points when training reasoning models with RL: cross-domain conflicts between math and code data, and training inefficiency caused by batches full of easy problems that produce near-identical rewards and zero useful gradient signal.

The first pain point, the team solves by splitting training into two stages. Stage one feeds the model exclusively hard math problems, coaxing out behaviors like backtracking, self-correction, and step-by-step decomposition. Stage two layers in code data, building on that reasoning skeleton to develop structured programming skills. Mixing math and code from the start, they found, produced shorter and shallower responses in both domains—math rewarded verbose chains of thought while code rewarded terse output, and neither got what it needed.

The second pain point—wasted compute on trivially solved samples—gets addressed through "history resampling." At the end of each training epoch, SRPO drops problems where every rollout was correct, since those generate no useful learning signal. It keeps problems with mixed outcomes or all-incorrect outcomes, the latter on the theory that earlier failures may become tractable as the policy improves. The result is a curriculum-style progression that keeps gradient updates meaningful throughout training.

On benchmarks, SRPO-Qwen-32B scored 50 on AIME24 and 41.6 on LiveCodeBench, both surpassing DeepSeek-R1-Zero-32B. The model also exhibited emergent reflective behaviors—self-verification, hesitation, and exploration—that grew more frequent as training progressed. For builders looking to train capable reasoning models without burning massive compute budgets, SRPO offers a concrete and now open-sourced recipe worth studying.