This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Multi-Agent Reinforcement Learning, Communication Scheduling, Large Scale
Communication can improve coordination in partially observed multi-agent reinforcement learning (MARL), but learning *when* and *who* to communicate with requires choosing among many possible sender-recipient pairs, and the effect of any single message on future reward is hard to isolate. We introduce **SCoUT** (**S**calable **Co**mmunication via **U**tility-guided **T**emporal grouping), which addresses both these challenges via temporal and agent abstraction within traditional MARL. During training, SCoUT resamples *soft* agent groups every $K$ environment steps (macro-steps) via Gumbel-Softmax; these groups are latent clusters that induce an affinity used as a differentiable prior over recipients. Using the same assignments, a group-aware critic predicts values for each agent group and maps them to per-agent baselines through the same soft assignments, reducing critic output complexity and variance. Each agent is trained with a three-headed policy: environment action, send decision, and recipient selection. To obtain precise communication learning signals, we derive counterfactual communication advantages by analytically removing each sender's contribution from the recipient's aggregated messages. This counterfactual computation enables precise credit assignment for both send and recipient-selection decisions. At execution time, the grouping module and critics are discarded and only the per-agent policy is run, preserving decentralized execution. Experiments on homogeneous gridworld benchmarks show that SCoUT learns targeted communication and remains effective in large scenarios, while prior methods degrade as the population grows. Finally, ablations confirm that temporal grouping and counterfactual communication credit are both critical for scalability. Project website, videos and code: [https://scout-comm.github.io/](https://scout-comm.github.io/)
Manav Vora, Gokul Puthumanaillam, Hiroyasu Tsukamoto, and Melkior Ornik. "SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{vora2026scout,
title={SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning},
author={Manav Vora and Gokul Puthumanaillam and Hiroyasu Tsukamoto and Melkior Ornik},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}