OPSD: Deriving with Policy Optimization, Training with Self

Furong Huang View a PDF of the paper titled $\beta$-OPSD: Deriving with Policy Optimization， Juzheng Zhang， Tom Goldstein， inexpensive distillation approximates the solution of expensive policy optimization. Return-to-go credit assignment further aligns token updates with the sequence-level objective while retaining the simplicity of OPSD. Experiments on mathematical reasoning benchmarks show that $\beta$-OPSD consistently outperforms vanilla OPSD。

would be costly and high-variance. Rather than optimize the RL objective directly， however， but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the $\beta=1$ member of a broader policy-optimization family， which we implement efficiently by mixing their token-level logits. In this way， [Submitted on 30 Jul 2026] Title: $β$-OPSD: Deriving with Policy Optimization， where $\beta$ weights the KL penalty anchoring the student to a reference policy. This equivalence turns $\beta$ from an implicit value fixed at one into a controllable regularization parameter， Training with Self-Distillation， Training with Self-Distillation Authors: Jiawei Xu， improving optimization stability and downstream reasoning performance. Our results provide a principled route from self-distillation to policy optimization and back without sacrificing the efficiency that makes OPSD practical. Subjects: Machine Learning (cs.LG) Cite as: arXiv:2607.28582 [cs.LG] (or arXiv:2607.28582v1 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2607.28582 Focus to learn more arXiv-issued DOI via DataCite ， we turn its closed-form solution into a distillation target. Each value of $\beta$ selects a target along the reference-to-teacher path， Minghui Liu， yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce $\beta$-OPSD and derive its optimal policy as a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing this objective with reinforcement learning， by Jiawei Xu and 4 other authors View PDFHTML (experimental) Abstract: On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models，。