This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Reward Modeling, Robot Learning, Vision Language Models, Foundation Models
Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior to policy training. We propose Demo2Reward, a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3–10 trajectories) to reduce false positives while retaining sufficient true positives. Crucially, this requires no additional model training or computational resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function.
Christian Gumbsch, Leonardo Barcellona, Lennard Schuenemann, Platon Karageorgis, Andrii Zadaianchuk, Zehao Wang, Sergey Zakharov, Fabien Despinoy, Rahaf Aljundi, and Stratis Gavves. "From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{gumbsch2026from,
title={From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models},
author={Christian Gumbsch and Leonardo Barcellona and Lennard Schuenemann and Platon Karageorgis and Andrii Zadaianchuk and Zehao Wang and Sergey Zakharov and Fabien Despinoy and Rahaf Aljundi and Stratis Gavves},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}