This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Deep Reinforcement Learning, Preference-based Reinforcement Learning
Preference-based reinforcement learning (PbRL) enables agents to learn behaviors from human feedback without requiring predefined reward functions. However, existing PbRL methods struggle to explore the preference space effectively, often converging prematurely to suboptimal policies that capture only a narrow subset of human preferences. In this work, we identify and address this preference exploration problem through population-based methods. By maintaining a diverse population of agents, our approach achieves broader coverage of the preference space than single-agent methods. This diversity yields two benefits: it improves reward model learning, and it generates preference queries with clearly distinguishable behaviors, reducing ambiguity for human evaluators. Our experiments show that existing methods can become trapped in local optima, requiring excessive feedback. Their performance also degrades substantially when evaluators make errors on similar trajectories, a realistic condition that methods assuming perfect oracle feedback tend to overlook. In contrast, our population-based approach explores preferences more effectively in environments with complex reward structures and is considerably more robust to noisy evaluators.
Brahim Driss, Alex Davey, and Riad Akrour. "PB²: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{driss2026pb,
title={PB²: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning},
author={Brahim Driss and Alex Davey and Riad Akrour},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}