Iterative GRPO: Batch-Online Multi-Turn RL via Single-Turn RLHF

תקציר מקורי באנגליתarXiv:2511.21638v3 Announce Type: replace Abstract: Practical LLM agents often operate over multi-turn conversations where success is determined only after the full interaction ends. Most multi-turn RL methods train via on-policy rollouts, but unlike in single-turn RLHF, the policy cannot produce a trajectory alone, since an external environment must respond after each agent turn. For conversational agents, this environment is a user, but real users are generally unavailable inside the training loop and simulated users are difficult to build faithfully. Separately, real-world deployment is rarely fully online or fully offline. The common production pattern is called "batch online," where the current policy is deployed to collect a batch of interaction data, then retrained on that batch and