Reinforcement Learning Journal, vol. 5, 2024, pp. 2298–2319.
Presented at the Reinforcement Learning Conference (RLC), Amherst Massachusetts, August 9–12, 2024.
This paper introduces a new benchmark, the Cross-environment Hyperparameter Setting Benchmark, that allows comparison of RL algorithms across environments using only a single hyperparameter setting, encouraging algorithmic development which is insensitive to hyperparameters. We demonstrate that the benchmark is robust to statistical noise and obtains qualitatively similar results across repeated applications, even when using a small number of samples. This robustness makes the benchmark computationally cheap to apply, allowing statistically sound insights at low cost. We provide two example instantiations of the CHS, on a set of six small control environments (SC-CHS) and on the entire DM Control suite of 28 environments (DMC-CHS). Finally, to demonstrate the applicability of the CHS to modern RL algorithms on challenging environments, we provide a novel empirical study of an open question in the continuous control literature. We show, with high confidence, that there is no meaningful difference in performance between Ornstein-Uhlenbeck noise and uncorrelated Gaussian noise for exploration with the DDPG algorithm on the DMC-CHS.
Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, and Adam White. "Cross-environment Hyperparameter Tuning for Reinforcement Learning." Reinforcement Learning Journal, vol. 5, 2024, pp. 2298–2319.
BibTeX:@article{patterson2024cross,
title={Cross-environment Hyperparameter Tuning for Reinforcement Learning},
author={Patterson, Andrew and Neumann, Samuel and Kumaraswamy, Raksha and White, Martha and White, Adam},
journal={Reinforcement Learning Journal},
volume={5},
pages={2298--2319},
year={2024}
}