Agentic RAG: teaching models to run their own searches

The standard Retrieval-Augmented Generation (RAG) pipeline is a fixed two-step ritual: embed the query, pull the top passages, hand them to the model, read once, answer. It works well when the answer sits in a single document. It struggles the moment a question needs two or three hops, a follow-up query, or a decision about whether to search at all. A recent line of work attacks that limit directly, by training the model to drive retrieval itself rather than being fed a fixed context.

From retrieve-then-read to search-as-an-action

In a classic pipeline, retrieval is something that happens to the model. The generator never chooses what to look up; a separate component decides that once, before generation begins. That is fine for lookups, but it is a poor fit for the compositional questions that dominate real evaluation sets, where you have to find one fact, use it to form the next query, and repeat.

Agentic RAG reframes search as an action the model can take mid-reasoning. The model writes some chain-of-thought, decides it needs a fact, emits a search query inside a special token boundary, receives the retrieved passages back into its context, reads them, and continues. It can search again, or stop and answer. The retrieval loop is no longer external scaffolding written by an engineer; it is behaviour the model has learned.

This is a different idea from the critique-and-revise systems covered in our note on self-correcting RAG. Those methods take retrieved passages and judge, filter, or rewrite them to improve faithfulness. Agentic search is about when and what to retrieve in the first place, learned end to end rather than bolted on as a verification stage. The two are complementary, and a mature system may well use both.

Why reinforcement learning, and why now

You could try to teach this behaviour with supervised fine-tuning, showing the model curated traces of good search sessions. In practice that is expensive to annotate and brittle: it teaches the surface form of a good trajectory, not the judgement behind it. The interesting result of the past year is that you can teach it with reinforcement learning (RL) and a much cheaper signal, namely whether the final answer was correct.

Search-R1 (Jin et al., 2025) is the clearest statement of the approach. It trains an LLM with RL to interleave step-by-step reasoning with real-time queries to a search engine. The model generates text; when it emits a search call, the system runs the retrieval and injects the returned passages back into the rollout; generation continues. Crucially, the tokens that come from the retriever are masked out of the loss, so the model is optimised on its own reasoning and search decisions rather than penalised for text it did not write. The reward is outcome-based: a simple check on whether the final answer matches the gold answer, with no need for a hand-labelled process reward at each step. The framework is deliberately agnostic to the policy-optimisation method (it reports Proximal Policy Optimisation and Group Relative Policy Optimisation variants) and to the retriever.

R1-Searcher (Song et al., 2025) reaches a similar destination by a two-stage route. The first stage rewards the model simply for learning to invoke search at all, using a format-style incentive; the second stage rewards it for using the retrieved results to answer correctly. It too relies purely on outcome supervision, with no distilled cold-start traces and no process reward model. The takeaway shared across both papers is that a well-shaped final-answer reward is enough to elicit competent multi-step search, which is a strikingly cheap ingredient for a capability this useful.

What emerges: self-directed multi-hop behaviour

The behaviour these systems learn is more interesting than the mechanism. Trained only to get the answer right, models begin to decompose questions on their own. Faced with a query that needs a bridge entity, they retrieve the first fact, notice the gap, and issue a second query built from what they just found. They learn to stop searching once they have enough, and, less reliably, to search when their own parametric knowledge is thin. None of this is explicitly rewarded step by step; it falls out of optimising the outcome.

This matters for the questions that break naive RAG. Multi-hop sets require chaining evidence across documents, and a single retrieval pass systematically fails them because the second-hop query cannot be formed until the first hop is read. A model that controls its own search loop can, in principle, form that second query. It also handles a subtler case well: knowing when not to retrieve, so that a simple question does not trigger a wasteful and potentially distracting search.

ZeroSearch: cutting the cost of the training loop

RL rollouts are hungry. Training a search agent means issuing enormous numbers of queries, and if every one hits a live search API, the bill and the rate limits become the bottleneck. There is a second problem too: the documents a real engine returns are uncontrolled and noisy, which injects variance into training that has nothing to do with the model’s decisions.

ZeroSearch (Sun et al., 2025), from Alibaba’s Tongyi Lab, addresses both by removing the real search engine from the training loop entirely. A lightweight supervised fine-tune first turns an auxiliary LLM into a simulator that, given a query, produces plausible documents. During RL the policy model queries this simulator instead of the web. A curriculum schedule progressively degrades the quality of the simulated documents, forcing the policy to get better at reasoning over imperfect evidence rather than leaning on clean retrieval. The reported result is that a simulator of modest size matches a real search engine for training purposes, and a larger one can exceed it, at a fraction of the API cost. It is a reminder that the environment a search agent trains against is a design choice, not a fixed given.

The honest trade-offs

None of this is free, and practitioners should weigh the costs before reaching for an agentic setup.

Latency. Every search call is a round trip: generation pauses, retrieval runs, passages are injected, generation resumes. A multi-hop answer might involve three or four such loops in sequence. That is a real user-facing cost that a single-shot pipeline does not pay, and it compounds with reranking or any other per-hop processing.

Training complexity. RL over long, tool-interleaved rollouts is fiddly. You are managing token masking for retrieved content, credit assignment across many steps, and the usual instability of policy optimisation, all while the retriever adds non-determinism to every episode. This is considerably harder to stand up and debug than a supervised fine-tune, let alone a prompt.

Reward hacking. Outcome-only rewards are the strength and the weakness. Optimise purely for a string match on the final answer and the model may learn shortcuts: guessing without searching when it can, gaming the answer format, or exploiting quirks of the evaluation set rather than genuinely retrieving and reasoning. The reward is a proxy for what you want, and models are diligent about finding the gap between proxy and intent.

Evaluation is hard. Measuring these systems well is its own problem. Final-answer accuracy hides how the answer was reached, so a model can be right for the wrong reasons or retrieve well yet answer poorly. Judging the search trajectory (were the queries sensible, was the evidence sufficient) needs richer benchmarks than a single accuracy number. This is where purpose-built RAG evaluation sets earn their keep. FRAMES (Krishna et al., 2024) is well suited to this work: its questions each require integrating information from several Wikipedia articles, and it reports a large gap between naive single-pass prompting and multi-step retrieval, exactly the gap agentic search is meant to close. CRAG (Yang et al., 2024) adds pressure from a different direction, with questions spanning popular to long-tail entities and facts that change on timescales from years to seconds, which is precisely where a model needs to decide to go and look rather than trust its weights. Open-RAGBench offers a further openly licensed option for building repeatable harnesses.

Where this sits for a practitioner

Agentic search is not a replacement for a solid retrieve-then-read baseline; it is what you reach for when the questions genuinely need it. If your workload is single-hop factual lookup, a well-tuned pipeline with good chunking and reranking will be cheaper, faster, and easier to reason about. If your workload is compositional, multi-hop, or time-sensitive, the ability to let the model run its own searches starts to pay for its complexity.

Either way, the underlying corpus still decides the ceiling. A model that searches beautifully over a thin or poorly-licensed index will still answer badly, so the choices in choosing your first RAG dataset and across the wider dataset directory matter as much as the training recipe. The lesson of the past year is that the retrieval loop itself can now be learned rather than hand-built, and that a plain “get the answer right” reward is a surprisingly powerful teacher.