When to Retrieve?

This is a detailed slide-by-slide review of the paper When to Retrieve?, which was submitted to ACL 2024 but was ultimately rejected.

The paper questions the efficiency of the standard Retrieval-Augmented Generation (RAG) framework, which naively performs retrieval for every query. The authors propose an adaptive retrieval model (RET indicator) that dynamically decides, for each query, whether external knowledge retrieval is necessary. If the LLM’s parametric memory is sufficient, retrieval is skipped; otherwise, external information is fetched.

This is the title slide for my presentation on the paper “When to Retrieve?”, which I delivered at the Yonsei University Machine Learning Lab on February 28, 2025.

To set the stage, I’m introducing a real-world application I developed, “momugo,” a restaurant recommendation app. This slide shows the user input screen, where a user is asking for a recommendation for a quiet place to talk with a friend while enjoying soju and hot fish cake soup.

This slide demonstrates the detailed output of the “momugo” app. When a user clicks on a recommended restaurant, it displays more information, including relevant review snippets that match the user’s query, providing a justification for the recommendation.

Here, I’m breaking down the initial data filtering process in my application. The system first performs a primary filtering based on structured data like category and business hours. Then, it moves to a more detailed secondary filtering based on the specifics of the user’s natural language query.

This slide illustrates the core retrieval and generation pipeline. After the initial filtering, the user’s query is used to find the top 10 most similar review embeddings from a vector database. The top 3 restaurants are then selected, and an LLM generates a descriptive recommendation reason for each.

I’m detailing the “Detailed Filtering” step. An LLM is prompted with a system message to analyze the user’s query and extract specific requirements, such as “corkage available” or “pet-friendly,” into a structured JSON format.

This slide provides a high-level overview, separating the entire process into two main phases: Retrieval and Generation. The retrieval part finds the most relevant restaurants, and the generation part creates the user-facing explanation.

I’m explaining the embedding process for the retrieval phase. The descriptions and latest reviews for each restaurant are converted into a 1536-dimensional vector using OpenAI’s text-embedding-3-small model and stored in a Pinecone vector database.

This slide details the retrieval mechanism. We embed all reviews and the user’s query. Using LangChain, we retrieve the top 10 review vectors based on cosine similarity. Finally, we determine the top 3 restaurants by calculating the average vector similarity for each.

This slide outlines the generation phase. The top 3 restaurants identified during retrieval, along with their metadata and the original user query, are used to create three separate prompts for the LLM.

I’m showcasing the prompt engineering aspect for the generation phase. A structured prompt, including a system message and a human message with the query and restaurant context, is fed to the LLM to generate a tailored recommendation reason for each of the top 3 restaurants.

This slide concludes the overview of my application’s architecture. It shows how the generated recommendation reasons (from the LLM) and the filtered restaurant data are combined to produce the final, detailed output card shown to the user.

Shifting from my personal project to the main topic, this slide introduces the standard Retrieval-Augmented Generation (RAG) process, breaking it down into the ‘Retrieval’ and ‘Generation’ stages.

Here, I pose a critical question about the standard RAG framework: Is the naive approach of always retrieving information for every query the most effective or efficient method?

This slide frames the core problem investigated in the paper. I’m questioning the fundamental assumption of RAG: “Should we always retrieve, regardless of the query?” This sets the stage for a more nuanced approach.

I’m introducing the central concept of the paper: an adaptive retrieval model. The key idea is to use an indicator, which the authors call “RET,” to decide whether to retrieve external information or to answer directly from the LLM’s parametric memory.

Now that we’ve established the need for a decision mechanism, the key research question becomes: “Where to place a RET?” In other words, how does the model learn when it’s appropriate to trigger the retrieval step?

This slide presents the core hypothesis of the paper. The authors believe that retrieval is unnecessary for “popular entities” (information likely stored in the LLM’s parameters) but crucial for “non-popular entities.”

Building on the hypothesis, I’m highlighting the practical challenge: How can a model quantitatively identify if an entity is “popular”? The paper proposes a model that can learn this distinction to enhance the efficiency and accuracy of the RAG process.

This slide formally introduces the paper I’m reviewing: “When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively” by Labruna, Campos, and Azkune. They name their proposed model “ADAPT-LLM.”

I’m providing context with related works. This slide contrasts “Closed-book QA,” which relies solely on an LLM’s internal knowledge, with “Open-book QA,” which always uses an external information retrieval system.

This slide summarizes key issues in the field and their corresponding solutions. It covers the high cost of retraining models, the lexical gap in keyword-based retrieval, and the latency costs associated with retrieval, positioning an adaptive approach as a solution.

Continuing with the related works, this slide discusses the problem of tool overuse in models like Toolformer and the unrealistic nature of previous popularity-based retrieval methods. It positions ADAPT-LLM as a solution that provides a robust baseline for comparison.

Here, I’m outlining the four-step process of the ADAPT-LLM framework: 1) A query is sent to the model. 2) The model decides whether to retrieve or not. 3) If needed, it retrieves information. 4) It generates the final answer.

This slide focuses on the critical decision-making step. The central question is how the model determines whether to retrieve information. The paper’s answer is to fine-tune the LLM on a specially curated dataset to learn this behavior.

I’m illustrating the data preparation process for fine-tuning ADAPT-LLM. The process starts with QA data, which is fed to a base LLM. The model’s answers are classified as correct or incorrect, which then informs the creation of different types of training prompts.

This slide provides a concrete example of the data preparation pipeline. Using a QA pair about the capital of South Korea, I show how the query is extracted from the source data.

Continuing the example, this slide shows the base LLM generating both a correct answer (“Seoul”) and an incorrect answer (“Busan”) to the query, which is a crucial step for creating the fine-tuning dataset.

This slide details how the fine-tuning dataset, named DS_adapt, is constructed. Based on the LLM’s correctness, three types of prompts are generated: a direct answer for correct parametric knowledge, a <RET> token for incorrect answers, and a context-based prompt for incorrect answers.

This slide visualizes the final step of the model creation process. The DS_adapt dataset, with its varied prompt structures, is used to fine-tune a base LLM, resulting in the final ADAPT-LLM.

I’m presenting the headline results from the paper. The table shows that ADAPT-LLM, trained on both NQ and SQuAD datasets, outperforms both the NEVER RETRIEVE (NR-LLM) and ALWAYS RETRIEVE (AR-LLM) baselines in terms of accuracy on the PopQA test set.

This slide outlines the structure of the experiments section of my review. I will cover three key areas: comparison to baselines, the model’s ability to determine when context is needed, and a comparison with the state-of-the-art approach for the PopQA dataset.

I’m beginning the detailed experimental setup. This slide specifies the training datasets used for the fine-tuning process: Natural Questions (NQ) and Stanford Question Answering Dataset (SQuAD).

This slide specifies the base Large Language Model used in the experiments. The authors chose the Llama-2-7B model as the foundation for their fine-tuning.

Here, I specify the dataset used for evaluating the models at test time. All configurations are evaluated on the PopQA dataset, which is designed to test knowledge of popular entities.

I’m explaining how the baseline models are created for comparison. The NR-LLM (NEVER RETRIEVE) is fine-tuned only on prompts that require direct, parametric answers.

Similarly, this slide explains the creation of the AR-LLM (ALWAYS RETRIEVE) baseline. This model is fine-tuned exclusively on prompts that include retrieved context, forcing it to always rely on external information.

This slide recaps the training process for the main model, ADAPT-LLM. It is trained on the comprehensive DS_adapt dataset, which includes a mix of parametric, retrieval-triggering, and context-aware prompts.

I’m presenting the main results table again, this time to emphasize the direct performance comparison. ADAPT-LLM consistently achieves the highest accuracy, demonstrating the effectiveness of its selective retrieval strategy.

To provide more context on the datasets, this slide presents a table comparing the statistics of NQ, SQuAD, and PopQA, including the number of questions and the average length of questions and answers.

Now, I’m moving to the second part of the experimental analysis: evaluating ADAPT-LLM’s ability to correctly decide when to retrieve. This slide sets up the analysis of the model’s decision-making accuracy.

This slide presents a detailed breakdown of ADAPT-LLM’s performance. It analyzes the accuracy of the model in four scenarios: when it correctly decides to retrieve (and is given context), when it incorrectly decides not to retrieve (and isn’t given context), and so on.

I’m highlighting a key observation from the results table. The accuracy for questions where the model chose to retrieve (Acc. w/ context for (RET)) seems quite low, around 33%. This prompts a deeper investigation.

This slide provides an explanation for the previously noted low accuracy. The authors point out that the performance of the underlying Information Retrieval (IR) system itself was a limiting factor.

To support the claim about poor IR performance, this slide presents results from Table 4 of the paper. It shows a massive accuracy gap when using the gold standard passages versus passages retrieved by the Contriever system, confirming the IR bottleneck.

Despite the issues with the IR component, this slide shows that the model’s decision-making process is sound. The histograms show a clear inverse correlation between entity popularity and the usage of the <RET> token, confirming the model learned the core hypothesis.

I’m now moving to the final experimental comparison: ADAPT-LLM versus the previous state-of-the-art method for the PopQA dataset. This slide visually contrasts the two approaches, highlighting differences in training data and thresholding methods.

This slide presents the results of the state-of-the-art comparison. While the accuracy is comparable, I’m highlighting the authors’ claims that ADAPT-LLM is a more generalizable and efficient approach due to its lower reliance on the IR system and its independence from dataset-specific features like popularity scores.

To begin the conclusion, I’m bringing back the diagram illustrating the creation of the DS_adapt fine-tuning dataset. This serves as a reminder of the core technical contribution of the paper.

This slide recaps the fine-tuning process itself, showing how the diverse set of prompts from DS_adapt is used to train the base LLM into the final, adaptive ADAPT-LLM.

In my final slide, I’m summarizing the key takeaways from the paper. The experiments demonstrate that ADAPT-LLM is a robust model that successfully learns when to retrieve information, outperforming standard baselines and showing strong potential as a general, efficient approach to adaptive RAG.