The RLHF Book. Reinforcement learning from human feedback, alignment, and post-training LLMs

This book is dedicated to a key task in modern AI engineering — aligning models with human preferences. Reinforcement Learning from Human Feedback (RLHF) allows models to be made safer, more understandable, user-friendly, and precisely tailored to the developer’s specific style. In his book, Nathan Lambert combines philosophical and economic ideas with the fundamental mathematics and computer sciences of RLHF, offering a practical guide to applying these methods to your own models.

You will learn how modern models are trained on human preferences, how to collect and enhance large-scale preference datasets, and get a detailed explanation of the basic methods of training using policy-gradient algorithms. The book covers Direct Preference Optimization (DPO), direct alignment algorithms, simplified methods for fine-tuning per preferences, and explains how the evolution of RLHF has led to the emergence of a new approach — RLVR. The author examines industrial practices of post-training: training character and personality, using feedback from AI, complex quality assessment schemes, and modern recipes for combining instructional training with RLHF. Lambert shares real experience creating open models like Llama-Instruct, Zephyr, Olmo, and Tülu.

After ChatGPT became an industrial product thanks to RLHF, the technology rapidly spread. In this book, Nathan Lambert offers for the first time an inside look at modern RLHF pipelines, their advantages and trade-offs, supporting explanations with practical experiments and minimal implementations. Readers gain a comprehensive understanding of the foundations of RLHF, optimization methods, constitutional AI, synthetic data, and new approaches to model evaluation — as well as an insight into the unresolved issues that the community is working on today. The book helps readers join the forefront of those creating and aligning the next generation of models.