This chapter covers alignment after SFT: DPO for pairwise preferences and GRPO for programmatic rewards - the methods that turn a base model into something you can deploy behind a gate (Chapter 48). After supervised fine-tuning, language models often need to be aligned with human preferences. "Aligned" means: when the model has a choice between multiple plausible outputs, it picks the one humans prefer. This is a different task from supervised fine-tuning, which teaches the model what is correct but not what is preferred.
Figure 1. Concept map for this episode: fight, desk translation, what to lock.
The preference learning problem
Suppose you have pairs of outputs from the model for the same prompt, with a human annotator preferring one over the other. The goal is to update the model so that, on future prompts, it produces outputs more like the preferred one and less like the rejected one. The classical RLHF approach trains an auxiliary "reward model" to predict the preference, then uses standard RL (PPO) to optimise the language model against the reward model. This is computationally expensive and notoriously unstable.
Direct Preference Optimisation (DPO)
DPO, introduced in 2023, observed that the optimal RL solution has a closed-form relationship to the reference model and the preference data. This lets you skip the explicit reward model and optimise the language model directly via a loss that compares the model being trained against a frozen reference (pre-alignment) copy of itself: the log-probability ratio of the preferred output under the trained model versus the reference model, minus that same ratio for the rejected output, scaled by a temperature hyperparameter and passed through a sigmoid.
Inside the loss, one term asks how much more the trained model likes the preferred answer than the reference model did before alignment; the other asks the same question for the rejected answer. Their difference is the relative preference shift — positive once the model has moved toward favouring the preferred answer over the rejected one, negative otherwise, with the temperature term controlling how sharply the loss reacts to that shift. Wrapping the whole thing in a standard binary-classification loss turns alignment into one direct question: did the model correctly prefer the better answer? Answering that, and only that, is what makes DPO simpler than classical RLHF — there is no separate reward model to train, tune, or have go stale.
Group Relative Policy Optimisation (GRPO)
GRPO, introduced in 2024 by the DeepSeek team, is an alternative that uses groups of model outputs and a programmatic reward function (rather than human-labelled preference pairs). The reward function might check whether the output: - Produces valid JSON in the expected schema. - Mentions only tickers that actually exist. - Adheres to a required output length. - Aligns with the realised trading outcome (in a finance-specific application).
Worked picture
A prompt yields four completions with programmatic rewards 0.9, 0.3, 0.5, and 0.7. The group's mean reward is 0.6, so each completion's centred advantage - how far above or below that mean it landed - is +0.3, -0.3, -0.1, and +0.1 respectively: reinforce the 0.9 and 0.7 completions, suppress the 0.3 one, regardless of absolute scale drift as the model improves.
What breaks in production
Any RL-style method, including GRPO, is vulnerable to reward hacking: the model finds a way to maximise the programmatic reward without actually solving the intended task. Example: if the reward is "output is valid JSON," the model can learn to output an empty JSON object every time - perfectly valid JSON, completely useless. The discipline: design reward functions that cannot be trivially gamed. Combine multiple rewards (JSON validity AND content quality AND alignment with realised outcome). Use adversarial validation (does the model still produce good outputs on prompts it has never seen during reward optimisation?). Audit outputs periodically for "too good to be true" patterns. The reward function is a contract. If the contract has loopholes, a sufficiently capable model will find them. Better to write airtight contracts up front than to discover the exploitation after deployment.
Key idea
DPO aligns from pairwise preferences without a separate reward model; GRPO aligns from programmatic, checkable rewards in groups. Both need airtight reward contracts - finance favours GRPO when JSON validity and outcome alignment are machine-verifiable.