$\beta$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

תקציר מקורי באנגליתarXiv:2607.28582v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the $\beta=1$ member of a broader policy-optimization family, where $\beta$ weights the KL penalty anchoring the student to a reference policy. This equivalence turns $\beta$ from an implicit value fixed at one into a controllable regularization parameter, yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce $\beta$-OPSD and derive its optimal policy as a geometric interpolation between