RAG fundamentals (questions 1 to 10)
1. Explain the core problem that Retrieval Augmented Generation (RAG) aims to solve in the context of large language models (LLMs).
RAG addresses the limitations of LLMs regarding factual accuracy, recency of information, and hallucination by grounding their responses in external, verifiable knowledge. It enables LLMs to access and synthesize information beyond their training data, providing more reliable and up-to-date answers.
2. Describe the primary considerations when designing a document ingestion and chunking strategy for a RAG pipeline.
Key considerations include maintaining semantic coherence within chunks, managing chunk size to balance retrieval precision and recall, and handling document structure (e.g., headings, tables) to avoid splitting critical information. Overly small chunks can lose context, while overly large chunks can introduce irrelevant noise.
3. How do embedding models contribute to the retrieval phase of a RAG pipeline, and what is a common failure mode?
Embedding models convert textual data (chunks and queries) into dense numerical vectors, allowing semantic similarity to be measured mathematically. A common failure mode is using an embedding model that is not well-suited to the domain or type of query, leading to poor semantic matching and irrelevant retrieval results.
4. Discuss the trade-offs between different vector indexing strategies (e.g., HNSW, IVF_FLAT) in a production RAG system, considering factors like latency, recall, and memory footprint.
HNSW (Hierarchical Navigable Small Worlds) offers excellent recall and query speed but can have higher index build times and memory usage. IVF_FLAT (Inverted File Index with Flat quantizer) provides a good balance, often faster to build and more memory efficient than HNSW, but may sacrifice some recall or query latency depending on the number of clusters. The choice depends on specific latency requirements, dataset size, and acceptable recall degradation.
5. Explain the role of prompt assembly in a RAG pipeline and how it differs from traditional prompt engineering.
Prompt assembly in RAG involves dynamically constructing a prompt by integrating retrieved context documents alongside the user's query and system instructions. Unlike traditional prompt engineering which focuses on crafting a static, optimal prompt, RAG's assembly dynamically injects relevant information to ground the LLM's response.
6. How can a RAG pipeline be designed to generate accurate citations, and what are the challenges in ensuring their reliability?
Accurate citations can be generated by associating retrieved document chunks with their source metadata (e.g., URL, title, page number) and including this information in the prompt for the LLM to reference. Challenges include ensuring the LLM accurately attributes specific generated sentences to the correct source chunk, especially when synthesizing information from multiple sources, and preventing hallucinated citations.
7. What metrics would you use to evaluate the effectiveness of the retrieval component in a RAG pipeline, and why?
Key metrics include Recall@k (proportion of relevant documents retrieved within the top k results), Mean Reciprocal Rank (MRR) for ranking quality, and Precision@k. These metrics assess how well the system identifies and ranks relevant information, which is crucial for providing high-quality context to the LLM.
8. Describe a common end-to-end failure mode in a RAG pipeline where the individual components (chunking, embedding, retrieval, LLM) might seem to work correctly in isolation, but the overall system fails.
A common failure mode is 'lost in the middle,' where relevant information is retrieved and included in the context window, but the LLM fails to utilize it because it's buried among less relevant information or placed in a suboptimal position. This can happen even with good retrieval if the context window is large and the LLM struggles to prioritize information.
9. Beyond fixed-size or sentence-based chunking, what are more advanced chunking strategies, and when would you apply them?
Advanced strategies include recursive chunking, where documents are split by hierarchical delimiters (e.g., headings, paragraphs) before falling back to fixed sizes, and summary-based chunking, where small chunks are summarized and then embedded alongside their original, larger context. These are applied when preserving semantic hierarchy or providing multi-granularity context is critical for complex queries.
10. How might a RAG pipeline dynamically adapt its retrieval strategy based on the user's query or conversation history?
Dynamic adaptation could involve using a router LLM to classify query intent (e.g., factual, analytical, creative) and select different embedding models or knowledge bases accordingly. For conversational history, previous turns could be used to refine the current query, expand keywords, or re-rank retrieved documents to maintain coherence and context over time.
RAG retrieval quality (questions 11 to 20)
11. Explain the core concept of hybrid search in RAG and why it's often preferred over a single retrieval method.
Hybrid search combines multiple retrieval techniques, typically sparse (e.g., BM25) and dense (e.g., vector embeddings), to leverage their respective strengths. Sparse methods excel at keyword matching and rare terms, while dense methods capture semantic similarity. This combination improves both recall and precision by addressing different facets of query relevance, leading to a more robust context for the LLM.
12. Describe how BM25 calculates document relevance and identify a scenario where its performance might be suboptimal for an LLM-based RAG system.
BM25 assesses relevance based on term frequency in the document, inverse document frequency across the corpus, and document length normalization. It's effective for exact keyword matches but struggles with semantic similarity or synonyms. This means it might miss relevant documents that do not share exact terms with the query, leading to an incomplete or poor context for an LLM.
13. What is the primary purpose of a reranker in a RAG pipeline, and what is a key trade-off when choosing a reranking model?
The primary purpose of a reranker is to re-order the initial set of retrieved documents, prioritizing those most relevant to the query to improve the quality of context provided to the LLM. A key trade-off when choosing a reranking model is between its performance (e.g., using a large cross-encoder for higher accuracy) and its inference latency. More sophisticated models often require significantly more computational resources per document pair, impacting overall system response time.
14. In what specific scenarios would you implement query rewriting as part of a RAG pipeline, and what are the potential pitfalls?
Query rewriting is beneficial for ambiguous queries, conversational turns where implicit context needs to be made explicit, or when expanding acronyms or synonyms to improve retrieval recall. Potential pitfalls include introducing hallucinations or biases if the rewritten query deviates too much from the user's original intent. It can also increase latency if the rewriting model is complex or adds multiple inference steps.
15. How does metadata filtering contribute to retrieval quality in RAG, and what considerations are important when designing a metadata schema?
Metadata filtering enhances retrieval precision by allowing pre- or post-filtering of documents based on structured attributes like document type, author, or date range, ensuring only contextually appropriate documents are considered. When designing a schema, consider the granularity required for filtering, potential for future expansion, and how it aligns with anticipated user query patterns. An overly restrictive or poorly designed schema can inadvertently filter out relevant documents.
16. Discuss the challenges of integrating document-level permissions into a RAG system and how these challenges impact retrieval design.
Integrating document-level permissions requires ensuring that the retrieval system only returns documents the user is authorized to view, which is critical for data security and compliance. This often necessitates filtering retrieved documents based on user roles or attributes, either at the index level during query time or post-retrieval before context is sent to the LLM. This adds complexity to the retrieval pipeline and can introduce additional latency or require specialized indexing strategies.
17. Describe a strategy to maintain document freshness in a RAG system that processes rapidly changing information, and what are the associated system design implications?
To maintain freshness, a common strategy involves incremental indexing or real-time updates for critical data sources, coupled with a scheduled full re-indexing for less volatile content. System design implications include managing update queues to handle data ingestion efficiently and ensuring index consistency during updates. This might also involve using a multi-index approach, where a 'hot' index contains the freshest data for immediate queries, while a 'cold' index stores historical data.
18. When a RAG system provides a factually incorrect answer, what are the first two steps you would take to debug the issue?
The first step is to examine the retrieved documents to determine if the correct information was present but overlooked, or if irrelevant or incorrect documents were retrieved. This helps isolate whether the issue is with retrieval. The second step is to analyze the LLM's generation process, checking if it correctly synthesized information from the provided context or if it hallucinated despite having relevant context, indicating a generation-side problem.
19. Beyond checking retrieved documents and LLM generation, what are two deeper potential root causes for persistent poor RAG answers, and how would you investigate them?
Two deeper root causes could be a mismatch between the embedding model's understanding and the domain's terminology, or issues with the chunking strategy leading to incomplete context. To investigate the embedding model, analyze embedding space clusters for relevant versus irrelevant documents to identify semantic misalignments. For chunking, manually review problematic documents' chunks to see if critical information is split across boundaries or if chunks lack sufficient self-contained context.
20. How would you quantitatively measure the retrieval quality of a RAG system, specifically distinguishing between recall and precision in a production environment?
To quantitatively measure retrieval quality, one would use metrics like Mean Reciprocal Rank (MRR) or Normalized Discounted Cumulative Gain (NDCG) for overall ranking quality, requiring human-annotated relevance judgments on a test set. Recall@k is the proportion of relevant documents retrieved within the top k results, |relevant ∩ top-k| / |relevant|. Hit rate, or success@k, is the percentage of queries where at least one relevant document is found within the top k. For precision, measure the proportion of relevant documents among the top-K retrieved documents for a given query, using the same test set with known relevant documents.
RAG and LLM evaluation (questions 21 to 30)
21. Describe the trade-offs between using precision and recall for evaluating retrieval in a RAG system.
Precision measures the proportion of retrieved documents that are relevant, focusing on the quality of retrieved results. Recall measures the proportion of all relevant documents that were successfully retrieved, emphasizing completeness. High precision is crucial when the cost of false positives is high, while high recall is important when missing relevant information is detrimental.
22. How would you measure the faithfulness of an LLM's answer to the retrieved context in a RAG system?
Faithfulness can be measured by assessing whether the generated answer is factually supported by the retrieved documents. This can be done through automated metrics like NLI (Natural Language Inference) models that check for entailment between the answer and context, or by human evaluation to verify claims against the source material.
23. Explain the role of 'golden sets' in evaluating RAG systems and their limitations.
Golden sets are curated datasets containing question-context-answer triples used for quantitative evaluation. They allow for reproducible benchmarking of retrieval and generation components. However, creating comprehensive and unbiased golden sets is labor-intensive, and they may not fully capture the diversity of real-world queries or nuanced answer requirements.
24. What are the advantages and disadvantages of using an LLM-as-judge for evaluating RAG outputs?
LLM-as-judge offers scalability and cost-effectiveness compared to human evaluators for tasks like answer relevance or faithfulness assessment. However, it can suffer from biases, lack of true understanding, and sensitivity to prompt wording, potentially leading to inconsistent or unreliable evaluations.
25. How can you implement online metrics to monitor RAG system performance in production?
Online metrics involve tracking user interactions and feedback in real-time, such as click-through rates on retrieved documents, user ratings of answers, or explicit feedback mechanisms. These metrics provide continuous insights into user satisfaction and can flag performance degradation or emerging issues without requiring a full offline evaluation cycle.
26. What is a 'regression gate' in the context of LLM evaluation, and why is it important?
A regression gate is a threshold for key evaluation metrics that must be met before a new model version or system change can be deployed. It prevents the introduction of performance degradations in critical areas like answer relevance or faithfulness, ensuring that improvements in one aspect do not negatively impact others.
27. Discuss a scenario where answer relevance might be high but faithfulness is low in a RAG system.
This can occur when the LLM generates an answer that uses keywords and concepts from the retrieved documents, making it seem relevant on the surface. However, the answer might introduce fabricated details or misinterpret the context, leading to factual inaccuracies not supported by the source material.
28. How might you construct a golden set to specifically test the RAG system's ability to handle ambiguous queries?
To test ambiguity, the golden set should include questions with multiple plausible interpretations. For each question, you would define multiple correct answers corresponding to these interpretations and ensure the retrieved documents provide information relevant to each interpretation, allowing evaluation of the system's ability to disambiguate or provide comprehensive answers.
29. When using LLM-as-judge, what prompt engineering strategies can mitigate evaluation biases?
Strategies include providing clear rubrics and definitions for evaluation criteria, using few-shot examples of good and bad evaluations, and employing adversarial prompts to probe for weaknesses. Iteratively refining prompts based on observed biases and comparing LLM judgments against human judgments can also help calibrate the judge LLM.
30. What are the challenges of defining and measuring 'answer relevance' in a RAG system, especially for complex or subjective questions?
Answer relevance is challenging because it can be subjective and context-dependent. For complex questions, relevance might involve synthesizing information from multiple retrieved documents, and for subjective queries, it might require nuanced understanding of user intent rather than just factual accuracy. Automated metrics often struggle to capture these subtleties, necessitating human judgment.
LLM fundamentals (questions 31 to 40)
31. Explain the role of tokenization in LLMs and common algorithms.
Tokenization breaks down text into smaller units (tokens) for LLMs to process. Byte Pair Encoding (BPE) and WordPiece merge frequent pairs into subwords. Unigram prunes a large vocabulary and does not merge like BPE. SentencePiece is an implementation of BPE or Unigram, not a fourth algorithm. Modern subword tokenizers virtually eliminate true out-of-vocabulary (OOV) issues by representing unseen words as sequences of known subwords, and they also handle whitespace and Unicode normalization.
32. Describe the core mechanism of self-attention in Transformers and its computational cost.
Self-attention allows the model to weigh the importance of different tokens in the input sequence when processing a specific token. It computes Query, Key, and Value vectors for each token and uses their dot products to determine attention scores, enabling context-aware representations. Multi-head attention performs this process in parallel across different representation subspaces. However, its computational cost scales quadratically with sequence length.
33. What is the significance of the context window size in an LLM, and what are its limitations?
The context window defines the maximum number of tokens an LLM can consider at once. A larger window allows for understanding longer dependencies (e.g., 32k tokens vs. 2k tokens), but increases computational cost and memory requirements quadratically with sequence length due to the attention mechanism. FlashAttention is exact attention: it cuts memory to O(N) and improves IO, but the multiply count stays O(N^2). Performer and Longformer change the asymptotic attention pattern. Context window extensions may require model retraining or architectural changes.
34. How does the KV cache optimize inference for LLMs, and what are its benefits?
The KV cache stores the Key and Value vectors computed for previous tokens during autoregressive decoding. This avoids recomputing them for each new token, significantly speeding up inference by only processing the current token's Query against cached Keys and Values, reducing latency. The KV cache size implications interact with batch size during inference, and its memory layout is important for efficient hardware utilization.
35. Contrast sampling temperature and top-p (nucleus) sampling for controlling LLM output diversity.
Temperature scales the logits before softmax, making the distribution sharper (lower temp, less diverse) or flatter (higher temp, more diverse). Top-p sampling selects tokens from the smallest set whose cumulative probability exceeds 'p', dynamically adjusting the number of choices based on probability mass, offering fine-grained control over randomness. Top-k sampling is another method that selects from the 'k' most likely tokens.
36. What is LLM hallucination, and what are its primary causes and mitigation strategies?
Hallucination refers to the generation of factually incorrect or nonsensical information presented as true. It can stem from the model overgeneralizing from its training data or lacking sufficient grounding. Mitigation strategies include Retrieval Augmented Generation (RAG), prompting patterns like chain-of-thought, factuality-focused fine-tuning, and Reinforcement Learning from Human Feedback (RLHF).
37. Explain the fundamental difference between fine-tuning and Retrieval Augmented Generation (RAG).
Fine-tuning adapts an LLM's internal weights to a specific task or domain using labeled data. RAG augments the LLM's input with relevant information retrieved from an external knowledge base. The retrieval model architecture, latency considerations during retrieval, and the grounding process are key aspects of RAG, allowing the LLM to answer questions based on up-to-date or proprietary data without altering model weights.
38. Describe the core idea behind LoRA (Low-Rank Adaptation) for efficient LLM fine-tuning.
LoRA freezes the pre-trained model weights and injects trainable low-rank decomposition matrices into specific layers, typically attention layers. This drastically reduces the number of trainable parameters, leading to faster training and smaller checkpoint sizes compared to full fine-tuning. Typical rank values are small (e.g., 4, 8, 16), and training tricks like careful initialization and optimizer choice are often employed.
39. How can you mitigate hallucination in LLM outputs?
Mitigation strategies include grounding responses with retrieved factual information (RAG), using prompting techniques that encourage factual accuracy (e.g., chain-of-thought, self-consistency), implementing post-generation fact-checking pipelines, and fine-tuning on high-quality, factual datasets. Reinforcement Learning from Human Feedback (RLHF) and factuality-focused fine-tuning are also effective. Factual accuracy should be evaluated with natural language inference (NLI) or a fact checker. N-gram overlap such as BLEU or ROUGE misses unsupported facts.
40. What is a potential trade-off when increasing the context window size of an LLM?
While a larger context window allows the model to consider more information, it significantly increases computational complexity and memory usage, often quadratically with sequence length. This can lead to slower inference times and higher hardware costs, posing a practical limitation. Benchmarking with task-specific metrics is recommended when assessing context-window trade-offs.
LLM production engineering (questions 41 to 50)
41. How would you optimize LLM inference latency in a production environment?
Latency optimization involves techniques like model quantization to reduce model size and computational requirements, and efficient batching to process multiple requests simultaneously. We can also explore hardware acceleration using GPUs or specialized AI chips and optimize network communication for distributed inference. Measuring latency involves tracking p95 and p99 response times under realistic load conditions.
42. Describe strategies for managing and reducing LLM inference costs.
Cost reduction focuses on optimizing resource utilization and model efficiency. This includes using smaller, fine-tuned models where appropriate, implementing intelligent request batching, and leveraging spot instances for non-critical workloads. Regularly monitoring token usage and identifying opportunities for prompt optimization can also significantly lower costs.
43. When and how would you implement caching for LLM requests?
Caching is most effective for identical or highly similar prompts that are frequently encountered, such as common user queries or system commands. We can implement a key-value store where the prompt serves as the key and the model's response as the value. Cache invalidation strategies are crucial to ensure responses remain up to date, especially for dynamic content.
44. What are the challenges and solutions for generating structured output from LLMs?
LLMs often produce freeform text, making structured output generation challenging. Techniques include using specific prompt engineering with examples, employing JSON schema constraints, or fine-tuning models on structured data. Validation after generation is essential to ensure adherence to the desired schema, catching errors like missing fields or incorrect data types.
45. Explain the concept of guardrails in LLM applications and how you would implement them.
Guardrails are mechanisms to ensure LLM outputs are safe, relevant, and adhere to predefined policies. Implementation can involve input filtering to block malicious prompts, output moderation to detect and reject harmful content, and using LLMs themselves as classifiers to check for policy violations. A common failure mode is over-blocking legitimate requests or under-blocking harmful ones.
46. How do you defend against prompt injection attacks in LLM applications?
Prompt injection attacks aim to manipulate LLM behavior through crafted inputs. Defenses include input sanitization to remove or neutralize potentially malicious instructions, using separate LLM calls for user input processing and instruction execution, and implementing strict output validation. It's an ongoing arms race, requiring continuous monitoring and adaptation of defense strategies.
47. Discuss the role of agents and tool calling in LLM production systems.
Agents enable LLMs to interact with external tools and environments, expanding their capabilities beyond text generation. Tool calling allows the LLM to decide when and how to use specific functions or APIs to gather information or perform actions. This requires careful design of tool descriptions and robust error handling for tool execution failures.
48. What key metrics would you monitor for an LLM in production, and why?
Key metrics include inference latency (p95/p99), throughput, error rates (including hallucinations and refusals), cost per token, and user satisfaction scores. Monitoring these helps identify performance bottlenecks, cost inefficiencies, and potential model degradation or safety issues. Drift detection for input and output distributions is also vital.
49. Describe a scenario where you might need to migrate an LLM in production and the considerations involved.
Migration might be necessary due to model performance improvements, cost reductions, or deprecation of an older model. Key considerations include ensuring backward compatibility of the API interface, performing A/B testing or shadow deployments to validate the new model's behavior, and managing the transition of any fine-tuned weights or embeddings. A rollback strategy is essential.
50. How do you handle model drift and ensure consistent performance over time?
Data drift means P(X) changes, and concept drift means P(Y|X) changes. A change in the statistical properties of the input data is data drift. Monitor input distributions and output quality against a baseline, and retrain when you measure a quality drop.
Get the free 50 RAG and LLM interview questions
Join the voluntary email list and open the free set of 50 RAG and LLM interview questions with short model answers. You will also get occasional new question sets. No spam, and you can unsubscribe at any time by replying "unsubscribe" to any email.
Each topic page has 14 to 18 questions with answer outlines: see all interview topics.
The 0 to 1 AI Engineer Interview Playbook covers answer frameworks, system-design walkthroughs and an 8-week plan. $19.99 on Gumroad.
Get the playbook