This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: multi-agent reinforcement learning, cooperation, Dec-POMDP
Centralized training with decentralized execution (CTDE) is a dominant paradigm for cooperative multi-agent reinforcement learning. Most successful value-based CTDE methods rely on value factorization, decomposing a joint value function into per-agent utilities under structural constraints such as additivity or monotonicity. While effective, these utilities are architectural components of a mixing network and lack a principled interpretation as standalone value functions. Moreover, existing approaches provide limited insight into the convergence properties of decentralized learning dynamics. We propose Decentralized Asymmetric Deep Q-Networks (Dec-ADQN), which decouples decentralization from structural value factorization. During training, a centralized, stateful action-value function is learned and used as a regression target for independently trained per-agent Q-functions that are executed in a decentralized manner. We show that these per-agent Q-functions approximate the conditional expectation of the centralized evaluator given each agent’s local information. Under exact optimization in a finite Dec-POMDP, we prove that Dec-ADQN implements a form of decentralized asymmetric policy iteration and converges to person-by-person optimal policies. Empirically, Dec-ADQN performs well across both independently decomposable tasks and tasks requiring tight coordination in the OvercookedV2 and SMAX benchmarks. These results suggest that projection-based decentralized learning offers an interpretable and theoretically grounded alternative to structural value factorization in cooperative MARL.
Rupali Bhati, Anurag Kadkol, Andrea Baisero, and Christopher Amato. "Decentralized Asymmetric DQN: Decentralization without Factorization in Multi-Agent Reinforcement Learning." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{bhati2026decentralized,
title={Decentralized Asymmetric DQN: Decentralization without Factorization in Multi-Agent Reinforcement Learning},
author={Rupali Bhati and Anurag Kadkol and Andrea Baisero and Christopher Amato},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}