Four Ways to Teach an AI to Draw a Cat

Same student, same cat, four very different lessons — and every way of training a language model is one of them. The cartoon is on top, the loss function underneath.

Trace the teacher SFT

supervised fine-tuning

The teacher draws a cat. You copy it, stroke for stroke.

you hear back nothing — just copy

You never practise on a cat of your own.

what's actually happening

Plain next-token prediction on curated (prompt, expert answer) pairs — the same objective as pretraining, pointed at better data. Under teacher forcing the model always conditions on the expert's prefix, so it only ever learns what to do in states the expert visits. At inference it conditions on its own tokens, drifts into states it never trained on, and errors compound. That's exposure bias.

dataExpert demonstrations, from humans or a stronger model
signalDense — a gradient on every token
distributionOff-policy: the expert's trajectories, not yours
fails whenThe student's own mistakes take it off the expert's path

Pick the better one DPO

direct preference optimization

You draw two cats. The teacher points at one: that one.

you hear back better, or worse

One nod for a whole drawing.

what's actually happening

Classic RLHF trains a separate reward model on preferences, then optimizes it with PPO. DPO skips both. Under a KL-constrained RL objective the optimal policy has a closed form, so the reward can be rewritten as an implicit log-ratio against a frozen reference model — turning the whole thing into one classification loss on preference pairs. Raise the likelihood of the winner, lower the loser, and let β control how far you may drift from the reference.

data(prompt, chosen, rejected) triples, usually collected once
signalRoughly one bit per pair, smeared over every token
distributionOff-policy: someone else's two answers, not fresh samples
fails whenThe winning answer is mostly bad, or the pairs go stale as the policy moves

Just a score RL

reinforcement learning · ppo, grpo

You draw a cat. The teacher writes 7 and walks away.

you hear back one number

Which line earned the 7? Good luck.

what's actually happening

The model samples its own rollouts, a verifier or reward model returns a scalar, and a policy gradient pushes up the log-probability of whatever led to a good score. Finally on-policy — it learns in the states it actually reaches. But one number has to be divided among thousands of tokens, which is a brutal credit assignment problem: high variance, tamed with advantage baselines (GRPO uses the mean over a group of rollouts) and a KL leash to the reference model.

dataJust prompts — plus a verifier or a learned reward model
signalSparse: a few bits per rollout, however long it was
distributionOn-policy: the model's own samples
fails whenRollouts get expensive, or the reward model is gameable

Notes on every line OPD

on-policy distillation

You draw a cat. The teacher fixes each stroke as you go.

you hear back a note on every stroke

Your cat, your mistakes, marked exactly where you made them.

what's actually happening

The student samples the trajectory, so the states are its own — but the teacher then scores every token by running one forward pass over that trajectory and handing back its full next-token distribution. The loss is the per-token gap between the two distributions. That's SFT's dense gradient on RL's on-policy states, with no reward model and no verifier. Reverse KL is mode-seeking, which pushes the student onto one teacher behaviour rather than blurring across all of them.

dataPrompts, plus a teacher model you can query for logits
signalDense — a full distribution at every single token
distributionOn-policy: student's rollouts, teacher-graded
fails whenNo teacher exists that's already better at the task