Promptriever - Instruction-Trained Retrievers

I’m sharing my full slide-by-slide review of the paper Promptriever: Instruction-Trained Retrievers, which was presented at ICLR 2025.

The paper tackles a big problem in current search models: they often fail to understand complex requests, especially negative ones (like “not A, but B”). The authors’ solution is Promptriever, a new model trained with a special dataset that forces it to actually follow instructions.

This is my presentation on “Promptriever: Instruction-Trained Retrievers,” which I put together for the Information Theory and Machine Learning Lab.

First, let’s acknowledge the researchers. The lead author is Orion Weller, who is affiliated with Johns Hopkins and Samaya AI. It’s also worth noting this work was accepted as a poster at ICLR 2025.

I want to start with the paper’s core claim, which I think is really powerful. The authors state that their new training method is the first to prove that search models can be “intelligent, instruction-following partners, not just data finders.”

Here are the other co-authors who contributed to this research.

I’ve structured this review into three parts: Motivation, the Promptriever model, and the Experiments. We’ll start with the motivation behind the research.

So, how do current search engines “think”? They mostly use a retriever based on semantic similarity to rank documents. On the surface, this seems fine, but what’s the problem?

This example makes the problem obvious. Imagine you need a laptop that is not a MacBook and costs under $1000. A standard retriever sees the keywords “MacBook” and “under $1000” in an article about the MacBook Air and incorrectly flags it as highly relevant.

This leads to a frustrating user experience. You’re forced to keep tweaking keywords and filters just to find what you want.

This is where Promptriever comes in. It doesn’t just use semantic similarity. Instead, it operates on “dynamic relevance definitions,” which allows for a much more intelligent process.

Let’s look at the same query again, but with Promptriever. It correctly understands the instructions—the core topic, the exclusion of MacBooks, and the price constraint. Because of this, it successfully returns a relevant document about a Dell XPS 13.

The key idea here is that Promptriever “dynamically adjusts relevance based on your natural language instructions.” It’s not just matching words; it’s understanding commands.

We’ve seen what it does, which leads to the next question: “But how on earth was this made?” Let’s get into the technical details.

Now, we’ll dive into the second section, where I’ll break down the Promptriever model’s architecture and training.

The architecture is a combination of the LLaMA-2 7B language model and a Bi-encoder.

The main technical hurdle they faced is a well-known one: standard fine-tuning for information retrieval often destroys a model’s instruction-following ability. So how did they keep the model intelligent?

The answer, as they stated in their core message, lies in their “novel training data, which makes ignoring commands impossible for correct answers.”

For context, standard retrieval models are trained on simple (Query, Document) pairs from datasets like MSMARCO.

Promptriever, however, uses a much richer format: Query + Instruction paired with Synthetic documents. This is what lets them train the model on complex, instruction-based prompts.

The most clever part of their training data is the Instruction-Negative. This is a document that’s correct for the query alone, but becomes incorrect when the instruction is added. This is what forces the model to pay attention.

Here’s a perfect example. For the query “What is the capital of France?,” a general article about Paris is a good result. But if you add the instruction “mention its average annual rainfall,” that article is now an instruction-negative, and a new document with rainfall data becomes the right answer.

Through this process, the model learns that if it ignores the instruction, it will retrieve the wrong results. To get the right answer, it has to carefully read and follow the command.

The authors were careful about quality control. They found that about 15% of their generated instructions made the original document irrelevant. For those cases, they used an LLM to generate a new, correct document as a substitute.

Creating these instruction-negatives was absolutely essential. Without them, the model could have just learned to ignore the instructions and still perform well on the base dataset. The negatives guarantee true instruction-following.

Now for the final section, “Experiments,” where we’ll look at the results.

For a fair comparison, they ran an “Apples-to-Apples” test against RepLLaMA, using the exact same data and hyperparameters.

They used a range of datasets and evaluated performance with metrics like NDCG@10, MRR, and importantly, p-MRR, which is designed to measure sensitivity to instructions.

The results were impressive. In short, Promptriever achieved state-of-the-art performance, showed better robustness, and could be improved zero-shot just by prompting.

This table gives a detailed breakdown. You can see that Promptriever gets high scores across the board, but it really shines in the p-MRR metric, which confirms its superior instruction-following ability.

On the in-domain MSMARCO dataset, the performance was on par with the strong RepLLaMA baseline. This is great because it shows that the model gained its new skills without sacrificing core retrieval performance.

This is where it gets interesting. When given a helpful prompt, Promptriever’s performance on out-of-domain datasets actually improves, while other models get worse. This proves that it is genuinely “promptable.”

This table looks at the standard deviation of scores across different prompts. Promptriever’s lower deviation means its performance is much more stable and consistent, regardless of how the query is phrased.

The ablation study confirms it all. The performance gains are a direct result of the instruction-based training with instruction-negatives, not because of other factors like longer queries.

The authors also proved their training “recipe” is general. It works well on other base models like Mistral and Llama 3, not just LLaMA-2. As they put it, “A golden recipe doesn’t discriminate against ingredients!”

Finally, let’s look at the reviewer feedback. When one reviewer claimed the data comparison was unfair, the authors argued that the data generation method is their core contribution. They also clarified that comparing to techniques like query rewriting was out of the paper’s scope.

In the end, the authors addressed all concerns. They ran the requested statistical tests and added more real-world examples during the rebuttal period, which satisfied the reviewers and got the paper accepted.