Reinforcement Learning Journal, vol. TBD, 2025, pp. TBD.
Presented at the Reinforcement Learning Conference (RLC), Edmonton, Alberta, Canada, August 5–9, 2025.
In settings where an AI agent nudges a human agent toward a goal, the AI can quickly learn a high-quality policy by modeling the human well. Despite behavioral evidence that humans hyperbolically discount future rewards, we model human as Markov Decision Processes (MDPs) with exponential discounting. This is because planning is difficult with non-exponential discounts. In this work, we investigate whether the performance benefits of modeling humans as hyperbolic discounters outweigh the computational costs. We focus on AI interventions that change the human's discounting (i.e. decreases the human's ""nearsightedness"" to help them toward distant goals). We derive a fixed exponential discount factor that can approximate hyperbolic discounting, and prove that this approximation guarantees the AI will never miss a necessary intervention. We also prove that our approximation causes fewer false positives (unnecessary interventions) than the mean hazard rate, another well-known method for approximating hyperbolic MDPs as exponential ones. Surprisingly, our experiments demonstrate that exponential approximations outperform hyperbolic ones in online learning, even when the ground-truth human MDP is hyperbolically discounted.
Ian M Moore, Eura Nofshin, Siddharth Swaroop, Susan Murphy, Finale Doshi-Velez, and Weiwei Pan. "When and Why Hyperbolic Discounting Matters for Reinforcement Learning Interventions." Reinforcement Learning Journal, vol. TBD, 2025, pp. TBD.
BibTeX:@article{moore2025when,
title={When and Why Hyperbolic Discounting Matters for Reinforcement Learning Interventions},
author={Moore, Ian M. and Nofshin, Eura and Swaroop, Siddharth and Murphy, Susan and Doshi-Velez, Finale and Pan, Weiwei},
journal={Reinforcement Learning Journal},
year={2025}
}