This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: multi-agent reinforcement learning, policy gradient, adaptive sampling, on-policy
Independent on-policy policy gradient algorithms are widely used for multi-agent reinforcement learning (MARL) in cooperative and no-conflict games, but they are known to converge sub-optimally when each agent’s individual policy gradient points away from an optimal joint equilibrium. Going beyond prior work, we observe that sub-optimal convergence can still arise \textit{even when the expected individual policy gradients of all agents point toward the optimal joint solution.} After collecting a finite set of trajectories, stochasticity in independent action sampling can cause the joint data distribution to deviate from the expected joint on-policy distribution. This \textit{sampling error} w.r.t. the joint on-policy distribution produces inaccurate gradient estimates that can make agents converge sub-optimally. We hypothesize that joint sampling error can be reduced through coordinated action selection and that doing so will increase the \textit{reliability} of policy gradient learning in MARL (i.e., the probability of agents converging to an optimal joint policy). To test this hypothesis, we first introduce an adaptive action sampling approach to reduce joint sampling error in the Centralized Training with Decentralized Execution setting. Our method, Cooperative Sampling Error Reduction (CoSER), continually adapts a centralized behavior policy to place higher probability on joint actions that are under-sampled w.r.t. the current joint policy. We then empirically evaluate CoSER on a diverse set of no-conflict multi-agent games and demonstrate that (1) CoSER reduces joint sampling error more efficiently than independent on-policy sampling and (2) this reduction increases the reliability of independent policy gradient algorithms.
Nicholas E. Corrado and Josiah P. Hanna. "Centralized Adaptive Sampling for Reliable Co-training of Independent Multi-Agent Policies." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{corrado2026centralized,
title={Centralized Adaptive Sampling for Reliable Co-training of Independent Multi-Agent Policies},
author={Nicholas E. Corrado and Josiah P. Hanna},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}