AI Engineer Interview Questions / Interview topics

Deep Learning Interview Questions (2026): 15 Questions with Answer Outlines

Deep learning interview questions check that you understand how neural networks train and why they fail. Expect questions on backpropagation, gradients, activation functions, optimizers, normalization, regularization, initialization and architecture choices. Use the framework below and connect each concept to a symptom you would see during training.

Get the playbook Free 50 questions

A 5-step way to answer any deep learning question

  1. Define it. One clear sentence.
  2. Explain the mechanism. The math idea in plain words.
  3. Name the symptom. What you would see in training curves if it goes wrong.
  4. Give the fix. The standard remedy and its trade-off.
  5. Tie to practice. When you have used or would use it.

Contents: Training fundamentals (questions 1 to 5) · Regularization and normalization (questions 6 to 10) · Architectures and practice (questions 11 to 15)

Training fundamentals (questions 1 to 5)

1. How does backpropagation work?

  • It computes gradients of the loss with respect to every parameter using the chain rule.
  • A forward pass computes outputs and loss; a backward pass propagates gradients layer by layer.
  • An optimizer then updates parameters using these gradients.
  • Automatic differentiation frameworks implement it.

Common pitfall: Confusing backpropagation (computing gradients) with the optimizer (updating weights).

2. What are vanishing and exploding gradients?

  • In deep networks, repeated multiplication can shrink gradients toward zero or grow them very large.
  • Vanishing gradients stall learning in early layers; exploding gradients cause unstable loss.
  • Fixes: residual connections, normalization, careful initialization, ReLU-family activations and gradient clipping.
  • RNNs suffer especially; LSTMs and GRUs were designed to help.

Common pitfall: Naming the problem without any fixes.

3. Why do we need nonlinear activation functions?

  • Without them, stacked linear layers collapse into one linear function.
  • ReLU is simple and avoids saturation for positive inputs.
  • GELU and SiLU are common in transformers.
  • Sigmoid and tanh saturate and can slow learning in deep networks.

Common pitfall: Saying ReLU never has problems; dead ReLUs can occur.

4. How do you choose a learning rate?

  • Too high causes divergence; too low trains slowly.
  • Use a learning rate finder or a small sweep.
  • Schedules such as warmup followed by cosine or linear decay are common.
  • The best learning rate depends on batch size and optimizer.

Common pitfall: Keeping a fixed learning rate without trying schedules.

5. What is the difference between SGD, Adam and AdamW?

  • SGD updates with the gradient, often with momentum.
  • Adam keeps per-parameter estimates of the first and second moments to adapt step sizes.
  • AdamW decouples weight decay from the gradient update, which regularizes more correctly than L2 inside Adam.
  • AdamW is a common default for transformers; SGD with momentum remains strong for some vision tasks.

Common pitfall: Saying AdamW is just Adam with L2 regularization.

Regularization and normalization (questions 6 to 10)

6. How do you detect and reduce overfitting?

  • Training loss keeps falling while validation loss rises.
  • Get more data or use augmentation.
  • Use regularization: weight decay, dropout, early stopping.
  • Reduce model capacity if needed.

Common pitfall: Judging only by training accuracy.

7. How does dropout work?

  • During training it randomly zeroes activations with some probability.
  • This discourages co-adaptation and acts like an ensemble.
  • At inference all units are used, with scaling handled by the implementation.
  • Large language models often use little or no dropout during pretraining.

Common pitfall: Forgetting that dropout behaves differently at training and inference time.

8. Batch normalization versus layer normalization?

  • Batch norm normalizes each feature across the batch; layer norm normalizes across features for each example.
  • Batch norm depends on batch statistics and behaves differently at inference.
  • Layer norm works with variable lengths and small batches, so transformers use it (or RMSNorm).
  • Both stabilize and speed up training.

Common pitfall: Saying transformers use batch norm.

9. Why does weight initialization matter?

  • Poor initialization can cause vanishing or exploding activations from the start.
  • Xavier (Glorot) initialization suits tanh-like activations; He initialization suits ReLU.
  • Residual networks often scale initialization of residual branches.
  • Good initialization speeds up convergence.

Common pitfall: Initializing all weights to zero, which makes units identical.

10. What is gradient clipping?

  • It caps gradient norm or values before the update.
  • It prevents rare large gradients from destabilizing training.
  • Common in RNNs and large transformer training.
  • It does not fix underlying data or learning rate issues.

Common pitfall: Using clipping to hide a learning rate that is far too high.

Free

Get the free 50 RAG and LLM interview questions

Join the voluntary email list and open the free set of 50 RAG and LLM interview questions with short model answers. You will also get occasional new question sets. No spam, and you can unsubscribe at any time by replying "unsubscribe" to any email.

We only use your email for this list. No results are promised; this is free practice material.

Architectures and practice (questions 11 to 15)

11. CNN, RNN or transformer: how do you choose?

  • CNNs use local receptive fields and weight sharing, good for images.
  • RNNs process sequences step by step but struggle with long dependencies and parallelism.
  • Transformers use attention, parallelize well and handle long-range dependencies, at quadratic attention cost.
  • Data size and latency limits also matter.

Common pitfall: Claiming transformers are always the best choice.

12. What are residual connections?

  • They add a layer's input to its output, so the layer learns a residual change.
  • They improve gradient flow and enable very deep networks.
  • Introduced in ResNet and used in transformers.
  • They make identity mappings easy to learn.

Common pitfall: Describing them only as a speed trick.

13. What is transfer learning?

  • Start from a model pretrained on large data and adapt it to a new task.
  • Fine-tune all or some layers depending on data size.
  • It reduces data needs and training time.
  • Watch for domain mismatch.

Common pitfall: Training from scratch on a small dataset.

14. How do you debug a model that does not learn?

  • Overfit a tiny batch first; if it cannot, there is a bug.
  • Check data loading, labels and preprocessing.
  • Check the learning rate, loss function and gradient norms.
  • Visualize predictions and training curves.

Common pitfall: Tuning hyperparameters before checking the data pipeline.

15. What is mixed precision training?

  • Using lower precision formats such as FP16 or BF16 for most operations.
  • It speeds up training and reduces memory on modern GPUs.
  • FP16 often needs loss scaling to avoid underflow; BF16 has a wider range.
  • Keep sensitive operations in higher precision as frameworks do.

Common pitfall: Assuming lower precision has no numerical risks.

Go deeper on related questions

FAQ

How much math is expected?

Be able to explain gradients, the chain rule and loss functions clearly; derivations are more common in research roles.

Are classic deep learning questions still asked for AI engineer roles?

Yes, often as warm-ups or for ML-heavy roles, even when the main focus is LLMs.

What should I review first?

Backpropagation, optimizers, overfitting and normalization come up most.

How do I practice?

Train a small network, break it on purpose (bad learning rate, no normalization) and explain the curves.

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers answer frameworks, system-design walkthroughs and an 8-week plan. $19.99 on Gumroad.

Get the playbook

All interview topics · Prep guides · Question deep dives · Free 50 RAG and LLM questions