This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Offline reinforcement learning, Bayesian inference.
Offline reinforcement learning (RL) seeks to improve a policy using only a fixed dataset of past interactions, without further environment exploration. To avoid overly optimistic decisions on uncertain or out-of-distribution actions, the learned policy should be supported by the data. Existing methods typically enforce this via heuristic modifications to objectives or value functions that encourage more conservative action selection. In contrast, we propose a principled alternative by introducing a conservative value prior, thereby modelling the belief that policies are expected to perform poorly unless the behavioural data provides evidence to the contrary. This yields a posterior that assigns high value only to supported actions, guiding the agent toward policies grounded in the data. Our approach thereby unifies Bayesian decision-making, uncertainty quantification and value regularisation while effectively mitigating distributional shift in offline RL. We develop this framework in a model-free setting with theoretical analysis on deterministic environments. We then present an exact inference algorithm for small-scale problems and finally extend it to a scalable deep learning variant compatible with standard off-policy algorithms. Our method achieves strong performance on benchmark locomotion tasks, outperforming comparable model-free baselines.
Filippo Valdettaro, Yingzhen Li, and Aldo A. Faisal. "Conservative Value Priors: A Bayesian Path to Offline Reinforcement Learning." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{valdettaro2026conservative,
title={Conservative Value Priors: A Bayesian Path to Offline Reinforcement Learning},
author={Filippo Valdettaro and Yingzhen Li and Aldo A. Faisal},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}