LLM Inference Optimization Interview Questions (2026): 15 Questions

A 5-step way to answer any inference optimization question

Name the metric. Time to first token, time per output token, throughput or cost per request.
Find the bottleneck. Compute bound (prefill) or memory-bandwidth bound (decode), or memory capacity for the KV cache.
Pick the lever. Batching, caching, quantization, speculative decoding, parallelism or a smaller model.
State the trade-off. Quality loss, tail latency, engineering cost or hardware needs.
Measure it. Benchmark on realistic prompt and output lengths at target concurrency.

Contents: Fundamentals (questions 1 to 4) · Batching and memory (questions 5 to 7) · Model-level techniques (questions 8 to 11) · Production trade-offs (questions 12 to 15)

Fundamentals (questions 1 to 4)

1. What are the prefill and decode phases?

Prefill processes the whole prompt in parallel and produces the first output token.
Decode generates one token at a time, each step reading the model weights and the KV cache.
Prefill is usually compute bound; decode is usually memory-bandwidth bound at small batch sizes.
Time to first token is driven by prefill; time per output token by decode.

Common pitfall: Treating generation as one uniform workload.

2. What is the KV cache and why does its size matter?

During decoding, keys and values for previous tokens are cached so they are not recomputed.
Its memory grows with batch size, sequence length, number of layers and KV heads.
KV cache memory often limits how many requests fit on a GPU at once.
Grouped-query attention, KV quantization and paged memory reduce its footprint.

Common pitfall: Forgetting that long contexts limit concurrency through KV cache memory.

3. Which latency metrics do you track for LLM serving?

Time to first token (TTFT) for perceived responsiveness.
Time per output token or inter-token latency for streaming speed.
End-to-end latency, throughput in tokens per second and requests per second.
Report p50 and p95 or p99, not just averages.

Common pitfall: Reporting only average latency.

4. Why is decoding memory-bandwidth bound?

Each decode step generates one token per sequence but must read all model weights from GPU memory.
With small batches there is little computation per byte read, so memory bandwidth limits speed.
Batching more sequences reuses each weight read across more tokens.
That is why batching and weight quantization speed up decode.

Common pitfall: Assuming a faster compute chip alone fixes decode speed.

Batching and memory (questions 5 to 7)

5. What is continuous batching?

Instead of waiting for a whole batch to finish, the server adds and removes sequences at each decode step.
It keeps the GPU busy when requests have different output lengths.
It greatly improves throughput over static batching in typical chat workloads.
Serving frameworks such as vLLM and TGI implement it.

Common pitfall: Describing static batching, where short requests wait for the longest one.

6. What problem does PagedAttention solve?

It stores the KV cache in fixed-size blocks, similar to virtual memory pages.
This reduces fragmentation and wasted reserved memory for variable-length sequences.
More sequences fit in memory, which raises throughput.
It also enables sharing KV blocks across requests with a common prefix.

Common pitfall: Confusing PagedAttention with a change to the attention math.

7. What is prefix (prompt) caching?

When many requests share the same prefix, such as a long system prompt, the KV cache for that prefix is reused.
It cuts prefill time and cost for repeated prefixes.
Put static content first and variable content last to benefit.
Several API providers offer discounted pricing for cached input tokens.

Common pitfall: Putting user-specific text before the shared system prompt.

Model-level techniques (questions 8 to 11)

8. How does quantization speed up inference, and what are the risks?

It stores weights (and sometimes activations or KV cache) in lower precision such as INT8 or 4-bit.
Smaller weights mean less memory and less bandwidth per decode step.
Post-training methods such as GPTQ and AWQ aim to keep accuracy at 4-bit.
Always re-run your task evaluation, since some tasks degrade more than benchmarks suggest.

Common pitfall: Assuming quantization is free because a generic benchmark barely moved.

9. What is speculative decoding?

A small draft model proposes several tokens; the large model verifies them in one forward pass.
Accepted tokens are kept; at the first rejection, resample that position from the normalized residual max(0, p-q).
With the standard acceptance rule, the output distribution matches the large model.
Speed-up depends on how often the draft is right, so it helps most on predictable text.

Common pitfall: Claiming speculative decoding changes output quality by design.

10. When would you use distillation or a smaller model instead?

If a smaller model meets quality on your evaluation set, it cuts latency and cost directly.
Distillation trains a small model to imitate a larger one on your task.
Routing sends easy requests to small models and hard ones to large models.
Validate on real traffic samples, not only public benchmarks.

Common pitfall: Optimizing serving tricks for a model that is larger than the task needs.

11. What are tensor parallelism and pipeline parallelism?

Tensor parallelism splits each layer's matrices across GPUs; it needs fast interconnect and reduces per-GPU memory and latency.
Pipeline parallelism places different layers on different GPUs and passes activations between them.
Pipeline parallelism can add bubbles unless micro-batching keeps stages busy.
Use them when the model or its KV cache does not fit on one GPU.

Common pitfall: Mixing up data parallelism (replicas) with model parallelism.

Production trade-offs (questions 12 to 15)

12. How do you trade off throughput and latency?

Larger batches raise throughput but can increase per-request latency.
Set an SLO, for example p95 TTFT, and maximize throughput subject to it.
Separate interactive and batch traffic so offline jobs do not hurt user latency.
Load test with realistic input and output length distributions.

Common pitfall: Optimizing throughput alone and breaking the user-facing latency target.

13. How do you reduce perceived latency in a chat product?

Stream tokens so users see output immediately.
Shorten prompts, cache shared prefixes and cap output length.
Use a faster model for simple turns and parallelize independent tool or retrieval calls.
Measure TTFT and full-response time separately.

Common pitfall: Ignoring time spent in retrieval or tool calls before generation even starts.

14. How do you estimate serving cost?

Cost depends on input tokens, output tokens, model size and hardware utilization.
For self-hosting, estimate tokens per second per GPU at target latency and divide GPU cost by throughput.
For APIs, multiply token counts by published prices and include retries.
Track cost per successful task, not just per call.

Common pitfall: Using peak throughput numbers that are not achievable at your latency target.

15. What would you check if latency suddenly doubled?

Compare input and output token lengths; prompt growth is a common cause.
Check queue depth, batch sizes and GPU memory pressure from long contexts.
Look for retries, upstream provider slowdowns or a model version change.
Use traces that break latency down by stage.

Common pitfall: Scaling hardware before finding which stage got slower.

Go deeper on related questions

FAQ

Do AI engineers need to know CUDA for these interviews?

Usually not. Most AI engineer loops expect conceptual knowledge of prefill, decode, KV cache and batching; kernel-level detail is for inference or ML systems roles.

Which serving frameworks should I know?

Know what vLLM, TGI or TensorRT-LLM provide at a high level (continuous batching, paged KV cache, quantization support) and name one you have used.

Is quantization always worth it?

It is often worth it for memory and speed, but always confirm task quality on your own evaluation set before shipping.

What is the fastest win in most products?

Shorter prompts, prefix caching, streaming and routing simple requests to a smaller model usually beat deeper infrastructure changes.

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers answer frameworks, system-design walkthroughs and an 8-week plan. $19.99 on Gumroad.

Get the playbook

All interview topics · Prep guides · Question deep dives · Free 50 RAG and LLM questions