Abstract
While originally developed for continuous control problems, Proximal PolicyOptimization (PPO) has emerged as the work-horse of a variety of reinforcementlearning (RL) applications including the fine-tuning of generative models.Unfortunately, PPO requires multiple heuristics to enable stable convergence(e.g. value networks, clipping) and is notorious for its sensitivity to theprecise implementation of these components. In response, we take a step backand ask what a minimalist RL algorithm for the era of generative models wouldlook like. We propose REBEL, an algorithm that cleanly reduces the problem ofpolicy optimization to regressing the relative rewards via a direct policyparameterization between two completions to a prompt, enabling strikinglylightweight implementation. In theory, we prove that fundamental RL algorithmslike Natural Policy Gradient can be seen as variants of REBEL, which allows usto match the strongest known theoretical guarantees in terms of convergence andsample complexity in the RL literature. REBEL can also cleanly incorporateoffline data and handle the intransitive preferences we frequently see inpractice. Empirically, we find that REBEL provides a unified approach tolanguage modeling and image generation with stronger or similar performance asPPO and DPO, all while being simpler to implement and more computationallytractable than PPO.