This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: policy gradient, reinforcement learning, gradient estimation, variance reduction, scaling
In deep learning, optimizers often limit effective step size through normalization or clipping, making update direction the key determinant of learning progress. Policy gradients scale each per-sample gradient term by an advantage estimate regardless of the action's probability: a rare action with large negative advantage can disproportionately influence the minibatch direction, even when the policy assigns it low mass. We introduce the \textit{Delightful Policy Gradient} (DG), which gates each per-sample term with a sigmoid of \emph{delight}, the product of advantage and the action's surprisal (negative log-probability). In tabular settings, we identify two mechanisms that improve learning. First, DG suppresses gradient variance from rare negative-advantage actions, improving directional accuracy. Second, when many decisions are trained jointly, it rebalances the expected gradient toward hard decisions, improving long-run progress. Both effects grow with the size of the state and action spaces. Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST, transformer sequence modelling, and continuous control, with a single temperature $\eta = 1$ across all experiments, and with larger gains on harder tasks.
Ian Osband. "Delightful Policy Gradient." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{osband2026delightful,
title={Delightful Policy Gradient},
author={Ian Osband},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}