This is the pre-proceedings for the RLC 2026. You may expect minor changes.

From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning

By Rafael Rodriguez-Sanchez, Cameron Allen, and George Konidaris

Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.

Will be presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.

Download:

Keywords: Factored MDPs, Disentanglement, Representation Learning, Reinforcement Learning

Abstract:

Algorithms that exploit factored Markov decision processes are far more sample‑efficient than factor‑agnostic methods, yet they assume a factored representation is known a priori---a requirement that breaks down when the agent sees only high‑dimensional observations. Conversely, deep reinforcement learning handles such inputs but cannot benefit from factored structure. We address this representation problem with Action‑Controllable Factorization (ACF), a contrastive learning approach that uncovers \emph{independently controllable} latent variables---state components each action can influence separately. ACF leverages sparsity: actions typically affect only a subset of variables, while the rest evolve under the environment's dynamics, yielding informative data for contrastive training. ACF recovers the ground‑truth controllable factors directly from pixel observations on three benchmarks with known factored structure---Taxi, FourRooms, and MiniGrid‑DoorKey---consistently outperforming baseline disentanglement algorithms. Moreover, we evaluate the quality of our learned representation for planning in Taxi and show empirically that ACF yields world models whose quality approaches that of expert-factored representations, bringing us closer to bridging the gap between deep RL and the factored MDP literature.

Citation Information:

Rafael Rodriguez-Sanchez, Cameron Allen, and George Konidaris. "From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning." Reinforcement Learning Journal, vol. 7, 2026, pp. TBD.

BibTeX:
@article{rodriguezsanchez2026from,
    title={From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning},
    author={Rafael Rodriguez-Sanchez and Cameron Allen and George Konidaris},
    journal={Reinforcement Learning Journal},
    volume={7},
    pages={},
    year={2026}
}