This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: exploration, average-reward, MaxEnt, entropy-regularization
Efficient exploration remains a central challenge in reinforcement learning, serving as a useful pretraining objective for data collection, particularly when an external reward function is unavailable. A principled formulation of the exploration problem is to find policies that maximize the entropy of their induced steady-state visitation distribution, thereby encouraging uniform long-run coverage of the state space. Many existing exploration approaches require estimating state visitation frequencies through repeated on-policy rollouts, which can be computationally expensive. In this work, we instead consider an intrinsic average-reward formulation in which the reward is derived from the visitation distribution itself, so that the optimal policy maximizes steady-state entropy. An entropy-regularized version of this objective admits a spectral characterization: the relevant stationary distributions can be computed from the dominant eigenvectors of a problem-dependent transition matrix. This insight leads to a novel algorithm for solving the maximum entropy exploration problem, EVE (EigenVector-based Exploration), which avoids explicit visitation estimation and instead computes the solution through iterative updates, similar to a value-based approach. To address the original unregularized objective, we employ a posterior-policy iteration (PPI) approach, which monotonically improves the entropy. We prove that EVE's core update equation converges linearly under standard assumptions. We empirically demonstrate that EVE produces policies with competitive exploration performance relative to rollout-based baselines in deterministic grid-world environments.
Jacob Adamczyk, Adam Kamoski, and Rahul V Kulkarni. "A Value-Based Approach to Maximum Entropy Exploration." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{adamczyk2026a,
title={A Value-Based Approach to Maximum Entropy Exploration},
author={Jacob Adamczyk and Adam Kamoski and Rahul V Kulkarni},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}