Abstract
Reinforcement learning (RL) for complex tasks remains a challenge, primarilydue to the difficulties of engineering scalar reward functions and the inherentinefficiency of training models from scratch. Instead, it would be better tospecify complex tasks in terms of elementary subtasks and to reuse subtasksolutions whenever possible. In this work, we address continuous spacelexicographic multi-objective RL problems, consisting of prioritized subtasks,which are notoriously difficult to solve. We show that these can be scalarizedwith a subtask transformation and then solved incrementally using valuedecomposition. Exploiting this insight, we propose prioritized softQ-decomposition (PSQD), a novel algorithm for learning and adapting subtasksolutions under lexicographic priorities in continuous state-action spaces.PSQD offers the ability to reuse previously learned subtask solutions in azero-shot composition, followed by an adaptation step. Its ability to useretained subtask training data for offline learning eliminates the need for newenvironment interaction during adaptation. We demonstrate the efficacy of ourapproach by presenting successful learning, reuse, and adaptation results forboth low- and high-dimensional simulated robot control tasks, as well asoffline learning results. In contrast to baseline approaches, PSQD does nottrade off between conflicting subtasks or priority constraints and satisfiessubtask priorities during learning. PSQD provides an intuitive framework fortackling complex RL problems, offering insights into the inner workings of thesubtask composition.