This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Multi-agent Reinforcement Learning, Reward Machines, Non-Markovian Rewards
Multi-agent general-sum stochastic games provide an effective mechanism for modelling many important real-world problems, but learning policies to maximize reward in this setting has received limited attention. Two significant challenges present themselves in this setting: (1) learning policies in general-sum games is difficult, and becomes even more challenging when the task and reward function structure are temporally extended, and (2) agents must contend with the non-stationarity of the environment resulting from the simultaneous actions of multiple agents. We address these challenges through two complementary mechanisms: reward machines, which capture non-Markovian reward functions using automata-based representations, and opponent modelling, which enables predicting the actions of other agents. We propose CROM and QROM, multi-agent Q-learning algorithms that exploit counterfactual experience learning over reward machine structure together with opponent modelling for general-sum stochastic games, where each agent knows its own reward machine a priori but not those of other agents. Theoretically, we establish convergence guarantees for our algorithms in tabular settings. Experimentally, we demonstrate that our algorithms learn effective policies with superior sample efficiency compared to existing baselines across diverse cooperative, competitive, and mixed-motive multi-agent environments.
Sriram Ganapathi Subramanian, Toryn Q. Klassen, and Sheila A. McIlraith. "Multi-Agent Reinforcement Learning with Reward Machines for Mixed Cooperative-Competitive Environments." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{subramanian2026multiagent,
title={Multi-Agent Reinforcement Learning with Reward Machines for Mixed Cooperative-Competitive Environments},
author={Sriram Ganapathi Subramanian and Toryn Q. Klassen and Sheila A. McIlraith},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}