Reinforcement Learning Journal, vol. 2, 2024, pp. 819–839.
Presented at the Reinforcement Learning Conference (RLC), Amherst Massachusetts, August 9–12, 2024.
Hierarchical Reinforcement Learning (HRL) approaches have shown successful results in solving a large variety of complex, structured, long-horizon problems. Nevertheless, a full theoretical understanding of this empirical evidence is currently missing. In the context of the *option* framework, prior research has devised efficient algorithms for scenarios where options are *fixed*, and the high-level policy selecting among options only has to be learned. However, the fully realistic scenario in which *both* the high-level and the low-level policies are learned is surprisingly disregarded from a theoretical perspective. This work makes a step towards the understanding of this latter scenario. Focusing on the finite-horizon problem, we present a meta-algorithm alternating between regret minimization algorithms instanced at different (high and low) temporal abstractions. At the higher level, we treat the problem as a Semi-Markov Decision Process (SMDP), with fixed low-level policies, while at a lower level, inner option policies are learned with a fixed high-level policy. The bounds derived are compared with those of state-of-the-art regret minimization algorithms for non-hierarchical finite-horizon problems, allowing to characterize when a hierarchical approach is provably preferable, even without pre-trained options.
Gianluca Drappo, Alberto Maria Metelli, and Marcello Restelli. "A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning." Reinforcement Learning Journal, vol. 2, 2024, pp. 819–839.
BibTeX:@article{drappo2024provably,
title={A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning},
author={Drappo, Gianluca and Metelli, Alberto Maria and Restelli, Marcello},
journal={Reinforcement Learning Journal},
volume={2},
pages={819--839},
year={2024}
}