כתבה
arXiv cs.CL ·
KL-Regularized Policy Optimization
On KL-Regularized Policy Optimization
אופטימיזציה של פוליצי KL-Regularized לאג'נטים של מודלי שפה גדולים.
תקציר מקורי באנגליתarXiv:2610.08963v1 Announce Type: cross Abstract: Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probabil
קרא במקור המקורי
arxiv.org
פתח כתבה מקורית