This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Reinforcement learning, reward shaping, constraint satisfaction, adaptive tuning
Temporal logic (TL) provides a flexible and expressive language for specifying complex task requirements that are often difficult to encode using scalar reward functions alone. Recently, there has been growing interest in reinforcement learning (RL) algorithms designed to solve tasks defined by TL specifications. However, many real-world applications require the simultaneous satisfaction of logical constraints and the optimization of environmental rewards, a setting in which TL alone is insufficient to capture both objectives. Consequently, it is essential to develop an RL framework capable of enforcing TL constraints while maximizing scalar returns, i.e. accumulated rewards, of the environment. This constrained optimization problem is particularly challenging in deep RL, where sparse TL satisfaction signals and competing environmental rewards often lead policies to disregard logical constraints. To address these challenges, we propose a novel adaptive critic shaping method to guide policy learning toward achieving both objectives. By decomposing TL constraints into sequential stages, the proposed method employs the max operator to directly shape the critic values of key states for achieving every stage, preventing the constraint satisfaction from being distracted by environmental rewards. Inspired by PID control, we further introduce an adaptive tuning mechanism that dynamically adjusts the shaped critic values, balancing the satisfaction of TL constraint and maximization of the environmental rewards. In addition, we formulate a sum-max RL framework and establish a theoretical guaranty on the convergence of the proposed method in stochastic environments. Empirical evaluations demonstrate that the proposed algorithm substantially outperforms existing reward-shaping methods in terms of both constraint satisfaction efficiency and environmental reward maximization.
Duo XU. "Adaptive Critic Shaping for Reinforcement Learning with Temporal Logic Constraint." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{xu2026adaptive,
title={Adaptive Critic Shaping for Reinforcement Learning with Temporal Logic Constraint},
author={Duo XU},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}