This is the pre-proceedings for the RLC 2026. You may expect minor changes.
Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Keywords: Successor Representations, World Value Functions, Grid Cells
World Value Functions (WVFs) provide a framework for learning goal-directed value functions that support super-exponential task composition at inference time. Yet, this composition requires a set of ``primitive'' tasks to compose that are learned through exhaustive interactions with all reward-relevant states. In this work, we significantly reduce the burden of this initial stage of learning. By applying the traditional Successor Representation (SR) factorisation of value into transition dynamics and reward we obtain a single domain-specific successor representation matrix, while each goal is encoded in a unique reward map. This significantly improves the transfer of learning between tasks as all domain-specific knowledge of the environment is shared by construction. Because reward maps are structurally simpler than value functions, which must encode information such as reachability, they are far more amenable to function approximation. Thus, inferring reward maps can be achieved efficiently through the inclusion of a vision module that predict reward locations. This provides a framework that constructs WVFs rather than learns WVFs monolithically through environment interactions. We validate this framework, called Successor Task Representations (STRs), in fully observable and partially observable 2D navigation, demonstrating that this decomposition matches oracle performance while classical RL and end-to-end deep RL baselines fail under changing reward structure. Finally, we show that by using STRs, hexagonal spatial representations emerge from goal-directed learning in sufficiently rich environments. This extends prior theoretical results from the SR which require random exploration and suggests that grid-cells, a hallmark of mammalian navigation, can emerge from goal-directed experience rather than unguided wandering or foraging behaviour.
Sergio Frasco, Devon Jarvis, and Geraud Nangue Tasse. "Learning World Value Functions with Successor Representation and Vision Models." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.
BibTeX:@article{frasco2026learning,
title={Learning World Value Functions with Successor Representation and Vision Models},
author={Sergio Frasco and Devon Jarvis and Geraud Nangue Tasse},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}