This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Learning from Human Feedback, Machine Teaching, Reward Learning
As autonomous agents are increasingly deployed across diverse operational contexts, aligning their behavior with human intent demands reward functions that remain robust to such changes rather than overfitting to any single operating condition. Inverse reinforcement learning (IRL) provides a principled way to infer such objectives from human feedback. However, existing machine teaching approaches for IRL focus on single-environment, demonstration-only settings, leaving underexplored how heterogeneous feedback modalities and environment dynamics jointly constrain reward functions that generalize across changing dynamics. Because demonstrations in one MDP entangle reward information with that environment’s specific transition structure, the resulting rewards frequently fail to generalize when the agent is deployed under different dynamics. We first analyze how feedback modalities constrain rewards, showing theoretically that, in the unlimited-data regime of an arbitrary MDP, pairwise comparisons impose strictly stronger global constraints than other modalities, and further analyzing the finite-budget regime where demonstrations impose tighter reward constraints. Beyond this theoretical analysis, we introduce a hierarchical machine teaching algorithm for IRL that operates across multiple MDPs. The algorithm first greedily selects informative environments that expose complementary reward constraints, then strategically queries low-cost feedback within those environments. Empirically, our method achieves substantially lower regret and stronger generalization to held-out environments than uniform teaching baselines under identical feedback budgets, demonstrating the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions.
Ali Larian, Qian Lin, Chang Zong Wu, and Daniel S. Brown. "Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{larian2026multimodal,
title={Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning},
author={Ali Larian and Qian Lin and Chang Zong Wu and Daniel S. Brown},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}