ML System Design Interview Questions (2026): 16 Questions with Answer Outlines

A 5-step way to answer any ML system design question

Frame the problem. Restate the business goal, turn it into an ML task (classification, ranking, regression) and agree on scale and latency.
Pick metrics. Name one offline metric, one online business metric and one guardrail metric that must not get worse.
Design data and features. Say where labels come from, how you avoid leakage and which features are computed online versus offline.
Start with a baseline. Propose a simple model first, then the upgrade path and what evidence would justify it.
Serve and monitor. Describe the serving path, the latency budget, rollout (shadow, canary, A/B) and drift monitoring.

Contents: Framing and metrics (questions 1 to 4) · Data, features and training (questions 5 to 8) · Serving and monitoring (questions 9 to 12) · Common design prompts (questions 13 to 16)

Framing and metrics (questions 1 to 4)

1. How do you start an ML system design interview?

Ask clarifying questions about the user, the goal, scale (users, QPS, catalog size) and latency limits before drawing anything.
Translate the goal into an ML objective, for example "rank posts by probability of a meaningful interaction".
State assumptions out loud and confirm them so the interviewer can redirect you early.
Sketch the high-level pipeline: data, training, serving, feedback loop.

Common pitfall: Jumping straight to a model architecture before agreeing on the objective and constraints.

2. How do you choose offline and online metrics?

Offline metrics match the task: precision and recall or AUC for classification, nDCG or recall@k for ranking, MAE or RMSE for regression.
Online metrics measure the business outcome, such as click-through rate, conversion, retention or fraud losses.
Add guardrail metrics like latency, complaint rate or diversity that must not regress.
Explain that offline gains do not always transfer online, so an A/B test is the final judge.

Common pitfall: Naming only accuracy, which is misleading on imbalanced data.

3. How do you handle class imbalance, for example in fraud detection?

Use metrics that reflect the rare class, such as precision-recall AUC and recall at a fixed precision.
Consider class weights, resampling or focal loss, and keep the evaluation set at the real class ratio.
Tune the decision threshold to the business cost of false positives versus false negatives.
Mention label delay: fraud labels such as chargebacks can arrive weeks later.

Common pitfall: Reporting high accuracy on a dataset where 99 percent of examples are negative.

4. What is the difference between offline evaluation and online A/B testing?

Offline evaluation scores a model on held-out historical data and is cheap and fast.
Online A/B testing exposes real users to the model and measures actual behavior change.
Offline results can be biased by the old model that generated the logged data (feedback loop and position bias).
Use offline evaluation to filter candidates and A/B tests to make the launch decision.

Common pitfall: Treating an offline metric win as proof that the product will improve.

Data, features and training (questions 5 to 8)

5. Where do training labels come from, and what can go wrong?

Implicit labels come from user behavior (clicks, purchases, watch time); explicit labels come from ratings or human annotators.
Implicit labels are noisy and biased by what the system already showed.
Human labels are costly and need clear guidelines and agreement checks.
Watch for label leakage, where a feature contains information only available after the outcome.

Common pitfall: Ignoring that clicks are biased by position and exposure.

6. What is training-serving skew and how do you prevent it?

It is a mismatch between features at training time and at serving time, which silently degrades predictions.
Causes include different code paths, stale features and time-travel leakage in training data.
Prevent it with shared feature definitions or a feature store and point-in-time correct joins.
Log the features actually used at serving time and train on those logs where possible.

Common pitfall: Recomputing features for training with a different pipeline than production uses.

7. How do you split data for evaluation in a time-dependent problem?

Split by time: train on the past and validate on a later period, mirroring production.
Random splits leak future information and overstate performance.
Keep users or sessions from appearing in both train and test when that would leak identity signals.
Use a rolling or backtest evaluation when seasonality matters.

Common pitfall: Using a random split on time-series or behavioral data.

8. Why propose a simple baseline first?

A baseline (heuristic, logistic regression, gradient-boosted trees) sets the bar any complex model must beat.
It is fast to ship, easy to debug and often strong on tabular data.
It exposes data and pipeline problems before you invest in deep models.
State the upgrade path and the evidence that would justify it.

Common pitfall: Starting with a large deep model without a reference point.

Serving and monitoring (questions 9 to 12)

9. How do you serve a model under a tight latency budget?

Split the budget across stages: feature fetch, candidate retrieval, model inference and post-processing.
Use precomputed features and embeddings, caching and batching where possible.
Use a multi-stage funnel: a cheap model narrows candidates and an expensive model ranks a short list.
Consider model compression such as distillation or quantization.

Common pitfall: Designing a single heavy model that scores every item for every request.

10. Batch prediction or real-time prediction: how do you decide?

Batch prediction precomputes scores on a schedule and is cheap and simple.
Real-time prediction uses fresh context such as the current session, at higher cost and complexity.
Choose by how fast inputs change and how much freshness improves the outcome.
Hybrid designs precompute heavy parts and add a light real-time layer.

Common pitfall: Choosing real-time inference by default when daily batch scores would work.

11. How do you roll out a new model safely?

Run in shadow mode to compare predictions without affecting users.
Move to a small canary, then an A/B test with predefined success and guardrail metrics.
Keep the previous model ready for fast rollback.
Version models, data and features so results are reproducible.

Common pitfall: Replacing the production model in one step with no rollback plan.

12. What do you monitor after launch?

System health: latency, error rate, throughput.
Data drift: changes in input feature distributions and missing values.
Prediction drift and, when labels arrive, model quality metrics.
Business metrics, with alerts and a retraining or rollback playbook.

Common pitfall: Monitoring only uptime and missing silent quality decay.

Common design prompts (questions 13 to 16)

13. Design a recommendation feed. What is the high-level architecture?

Candidate generation retrieves hundreds to thousands of items from millions using cheap methods such as two-tower embeddings.
A ranking model scores candidates with richer features to predict engagement.
A re-ranking step applies diversity, freshness and business rules.
Log impressions and outcomes to train the next model.

Common pitfall: Trying to rank the entire catalog with one expensive model.

14. Design a search ranking system.

Understand the query (spelling, intent, entities), then retrieve with lexical and semantic methods.
Rank with a learning-to-rank model using query, document and user features.
Evaluate offline with nDCG on judged queries and online with click and success metrics.
Correct for position bias when training on clicks.

Common pitfall: Training on raw clicks without accounting for position bias.

15. Design a content moderation system.

Combine rules, ML classifiers per policy and human review for uncertain cases.
Set thresholds per policy based on the cost of false negatives versus false positives.
Plan for adversarial adaptation and new abuse patterns.
Track recall at a fixed precision and reviewer workload.

Common pitfall: Using a single global threshold for all harm types.

16. How do you handle the cold-start problem?

For new users, use onboarding signals, popular items and context such as location or device.
For new items, use content features (text, image embeddings) instead of interaction history.
Use exploration, for example a bandit, to gather data on new items.
Measure cold-start performance separately from the overall metric.

Common pitfall: Saying "collaborative filtering" without explaining what happens for items with no interactions.

Go deeper on related questions

FAQ

How long is an ML system design interview?

Usually 45 to 60 minutes. Spend the first few minutes clarifying, most of the time on the design and the last minutes on monitoring and trade-offs.

Do I need to know specific model architectures?

Know a few strong defaults (gradient-boosted trees, two-tower retrieval, a deep ranking model) and why you would pick each. Clear trade-offs matter more than naming the newest architecture.

How is ML system design different from AI or LLM system design?

Classic ML system design centers on labels, features and trained models; LLM system design centers on prompts, retrieval, evaluation of generated text and cost per token. Many loops now include both.

What is the most common mistake?

Skipping metrics. Without an offline metric, an online metric and a guardrail, the interviewer cannot tell how you would know the system works.

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers answer frameworks, system-design walkthroughs and an 8-week plan. $19.99 on Gumroad.

Get the playbook

All interview topics · Prep guides · Question deep dives · Free 50 RAG and LLM questions