This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Multi-Agent Reinforcement Learning, Offline Reinforcement Learning, Diffusion
Offline multi-agent reinforcement learning (MARL) enables policy learning from fixed datasets, but is prone to $\textbf{coordination failure}$: agents trained on static, off-policy data converge to suboptimal joint behaviours because they cannot co-adapt as their policies change. We introduce $\textbf{CODA}$ ($\textbf{C}$oordination via $\textbf{O}$n-Policy $\textbf{D}$iffusion for Multi-$\textbf{A}$gent Reinforcement Learning}), a diffusion-based multi-agent trajectory generator for data augmentation that samples conditioned on the current joint policy, producing synthetic experience which better reflects the evolving behaviours of the agents, thereby providing a mechanism for co-adaptation. We find that previous diffusion-based augmentation approaches are insufficient for fostering multi-agent coordination because they produce static augmented datasets that do not evolve as the current joint policy changes during training; CODA tackles this by more closely simulating on-policy learning and is a meaningful step toward coordinated behaviours in the offline setting. CODA is algorithm-agnostic and can be layered onto both model-free and model-based offline reinforcement learning pipelines as an augmentation module. Empirically, CODA resolves canonical coordination pathologies in continuous polynomial games.
Marcel Hedman, Kale-ab Tessera, Juan Claude Formanek, Anya Sims, Riccardo Zamboni, Trevor McInroe, John Torr, and Elliot Fosong. "CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{hedman2026coda,
title={CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning},
author={Marcel Hedman and Kale-ab Tessera and Juan Claude Formanek and Anya Sims and Riccardo Zamboni and Trevor McInroe and John Torr and Elliot Fosong},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}