This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Foundations, Policy Improvement
We revisit a fundamental "chicken-and-egg" problem in reinforcement learning: policy improvement depends on the state visitation distribution of the updated policy, which is unknown at training time. Conservative updates solve this problem, but at the cost of shrinking the policy update. This paper explores an alternative solution, Approximate Next Policy Sampling (ANPS), which modifies the training distribution to approximately equal the next policy's state visitation distribution. To demonstrate the feasibility and efficacy of ANPS, we introduce Stable Value Approximate Policy Iteration (SV-API), which uses an iteratively updated behavioral policy to gather data, only committing to a new policy once a convergence condition has been met. The update is guaranteed to be safe if certain stability criteria are met, otherwise, it is no less safe than standard approximate policy iteration. We find the application of SV-API to PPO matches or improves performance on high-dimensional control (Atari) and continuous control benchmarks. Moreover, it executes substantially larger target policy updates. These results demonstrate the viability of ANPS as a new solution to a classic problem.
Dillon Sandhu and Ronald Parr. "Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{sandhu2026approximate,
title={Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL},
author={Dillon Sandhu and Ronald Parr},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}