This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: reinforcement learning, proximal policy optimization, asynchronous training, large
Decoupled PPO has been a successful reinforcement learning (RL) algorithm to deal with the high data staleness under the asynchronous RL setting. Decoupled loss used in decoupled PPO improves coupled-loss style of algorithms' (e.g., standard PPO, GRPO) learning stability by introducing a proximal policy to decouple the off-policy correction (importance weight) from the policy update constraint (trust region). However, the proximal policy requires an extra forward pass through the model at each training step, creating a computational overhead for large language models training. We observe that since the proximal policy only serves as a trust region anchor between the behavior and target policies, we can approximate it through simple interpolation without explicit computation. We call this approach A-3PO (APproximated Proximal Policy Optimization). A-3PO eliminates this overhead, accelerating training by 1.8$\times$ speedup while maintaining comparable performance. Code \& off-the-shelf example are contributed to the open-source RL training system AReaL: https://github.com/areal-project/AReaL/tree/v1.0.0.rc1.
Xiaocan Li, Shiliang Wu, and Zheng Shen. "A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{li2026a3po,
title={A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation},
author={Xiaocan Li and Shiliang Wu and Zheng Shen},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}