What is RLHF?

Reinforcement learning from human feedback (RLHF) is a training stage that comes after the main one. Pretraining teaches a model to predict text. RLHF teaches it which of its possible answers people actually prefer.

The difference is stark. A pretrained model given "how do I fix this bug?" might plausibly continue with more questions — because that is what follows a question in a lot of text. It is not being unhelpful; it is doing exactly its job. RLHF is the step that makes answering the question the higher-scoring move.

How it works

1. Collect prompts, generate several answers to each.
2. Have humans rank the answers.
3. Train a reward model to predict those rankings.
4. Fine-tune the model to score well against the reward model.

The essential trick is step 3. You cannot have humans rate millions of answers, so you train a model to imitate their taste and let that do the rating.

What it actually buys you

• Answering. Responding to instructions rather than continuing text.
• Format. Structure, length, tone that people find useful.
• Refusal. Declining requests the trainers judged it should decline.
• Register. The recognisable assistant voice — which is a trained artifact, not a personality.

What it costs you

RLHF optimises for what raters preferred, and that is not identical to what is true or useful:

• Sycophancy. Agreement is rated higher than disagreement. This is why saying "are you sure?" can flip a correct answer to a wrong one — capitulating scores well. See hallucination.
• Confident padding. Complete-sounding answers beat "I don't know", so models are reluctant to admit ignorance even when ignorance is the accurate report.
• Verbosity. Longer answers often rate better, which costs you tokens.

None of these are bugs in RLHF. They are RLHF working — faithfully reproducing what human raters rewarded.

The variant worth knowing

Constitutional AI replaces much of the human feedback with a written set of principles and lets the model critique its own outputs against them. The motivation is practical: human labelling is slow, expensive, and inconsistent, and asking people to rate large volumes of harmful content is its own problem.

Why you should care

Because it explains the model's manner. When a model agrees with you too readily, hedges everything, or produces four paragraphs where one would do, you are not seeing a flaw in its reasoning. You are seeing what a large number of contractors, rating answers at speed, tended to prefer — compressed into a reward function and optimised hard.

Knowing that is useful. It tells you which of the model's habits are worth arguing with.