Reinforcement learning from human feedback: humans compare model outputs. A reward model is trained to predict which output humans prefer. The policy is optimized to maximize the reward, classically with PPO. A KL penalty keeps the policy close to a reference model.

Direct Preference Optimization trains the policy directly on preference pairs. It derives a classification-style loss from the RLHF objective with a KL constraint. It needs no separate reward model and no online RL sampling during training. A reference model is still used in the loss.

RLHF and DPO Interview Questions (2026): 14 Questions with Outlines

A 5-step way to answer any post-training question

Place the stage. Pretraining, SFT, preference optimization or evaluation.
State the objective. What loss or reward is optimized.
Name the data. Demonstrations, comparisons or AI feedback.
Name the risk. Overfitting, reward hacking, mode collapse or forgetting.
Measure. Human preference win rates, benchmarks and safety tests.

Contents: Pipeline (questions 1 to 4) · DPO and alternatives (questions 5 to 8) · Data and evaluation (questions 9 to 14)

Pipeline (questions 1 to 4)

1. What are the main stages of post-training an LLM?

Supervised fine-tuning (SFT) on instruction and response demonstrations.
Preference optimization using comparisons, through RLHF or methods like DPO.
Safety tuning and evaluation.
Iterations often repeat these steps with new data.

Common pitfall: Treating instruction tuning and RLHF as the same thing.

2. What is RLHF?

Reinforcement learning from human feedback: humans compare model outputs.
A reward model is trained to predict which output humans prefer.
The policy is optimized to maximize the reward, classically with PPO.
A KL penalty keeps the policy close to a reference model.

Common pitfall: Saying humans score every training example online.

3. How is a reward model trained?

It takes a prompt and a response and outputs a scalar score.
It is trained on pairs, so the preferred response scores higher, often with a Bradley-Terry style loss.
It is usually initialized from the SFT model.
Its quality limits how well RLHF can work.

Common pitfall: Ignoring that reward models are imperfect proxies.

4. Why is a KL penalty used in RLHF?

It penalizes the policy for drifting too far from the reference (usually SFT) model.
This limits reward hacking and keeps language fluent.
Too strong a penalty limits improvement; too weak allows exploitation.
It is tuned as a hyperparameter.

Common pitfall: Not explaining what the KL is measured against.

DPO and alternatives (questions 5 to 8)

5. What is DPO?

Direct Preference Optimization trains the policy directly on preference pairs.
It derives a classification-style loss from the RLHF objective with a KL constraint.
It needs no separate reward model and no online RL sampling during training.
A reference model is still used in the loss.

Common pitfall: Saying DPO needs no reference model.

6. DPO versus PPO-based RLHF: what are the trade-offs?

DPO is simpler, more stable and cheaper to run.
PPO-based RLHF can use online samples and a reward model, which can help on some tasks.
DPO is limited to the preference data collected offline.
Many teams use DPO-style methods for simplicity.

Common pitfall: Claiming one is always better.

7. What is RLAIF or constitutional AI?

Using AI feedback instead of, or alongside, human feedback for preferences.
Constitutional approaches have a model critique and revise outputs against written principles.
It scales feedback collection at lower cost.
Quality depends on the judge model and principles.

Common pitfall: Assuming AI feedback has no biases.

8. What is reward hacking?

The policy finds outputs that score highly on the reward model but are not actually better.
Examples: overly long answers, flattery or exploiting reward model blind spots.
Mitigate with KL penalties, better reward models, length controls and human evaluation.
Monitor for drifts like growing response length.

Common pitfall: Trusting rising reward as proof of improvement.

Data and evaluation (questions 9 to 14)

9. How do you collect good preference data?

Use diverse prompts that reflect real usage.
Give annotators clear guidelines on helpfulness, honesty and safety.
Measure inter-annotator agreement and audit quality.
Include hard cases and refusals.

Common pitfall: Collecting comparisons only on easy prompts.

10. How do you evaluate an aligned model?

Human or model-judged pairwise win rates against a baseline.
Task benchmarks to check capabilities did not regress.
Safety and refusal tests, including over-refusal.
Real-user signals after launch.

Common pitfall: Using a single benchmark score.

11. What is the alignment tax?

Capability loss on some tasks after alignment tuning.
It can show up as worse benchmark scores or excessive refusals.
Mixing in pretraining or SFT data and careful KL tuning can reduce it.
Measure it explicitly.

Common pitfall: Ignoring capability regressions.

12. What is over-refusal?

The model refuses harmless requests that look similar to harmful ones.
It hurts usefulness and user trust.
Fix with better data covering borderline but safe requests.
Track refusal rates on benign test sets.

Common pitfall: Measuring only harmful-request refusal.

13. When would an application team use DPO?

To shape tone, format or preferences for a specific product using their own comparison data.
When prompting and SFT are not enough.
When they can collect reliable preference pairs.
Evaluate against the SFT model before shipping.

Common pitfall: Running DPO without a clear preference signal.

14. How does SFT differ from preference optimization?

SFT imitates demonstrations; it teaches format and basic behavior.
Preference optimization learns from comparisons and can push beyond demonstration quality.
SFT is usually done first to give a good starting policy.
Both can cause forgetting if overdone.

Common pitfall: Skipping SFT and applying preference tuning to a base model without explaining why.

Go deeper on related questions

FAQ

Do AI engineers need to implement RLHF?

Rarely. Most roles expect you to explain the pipeline and trade-offs; research roles may ask for details.

Is PPO still used?

PPO-based RLHF and newer RL methods are still used, especially at large labs; DPO-style methods are popular for their simplicity.

What should I remember about DPO?

It optimizes directly on preference pairs, needs no separate reward model and still uses a reference model.

How do I prepare?

Be able to draw the pipeline from pretraining to SFT to preference tuning and explain each loss in words.

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers answer frameworks, system-design walkthroughs and an 8-week plan. $19.99 on Gumroad.

Get the playbook

All interview topics · Prep guides · Question deep dives · Free 50 RAG and LLM questions