Agentic RAG vs Classic RAG: What Changes in the Architecture
Classic RAG retrieves once, in a fixed pipeline, whether or not retrieval is needed. Agentic RAG puts an agent in charge of planning, routing and checking retrieval. Here is the precise definition, what the agent actually adds, what it costs, and when a fixed pipeline is still the better engineering choice.
"Agentic RAG" is used loosely, so here is the definition this post uses, once. In the survey Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (Singh et al., arXiv 2501.09136, first submitted 15 January 2025), Agentic RAG means embedding autonomous AI agents into the RAG pipeline. Those agents use agentic design patterns — reflection, planning, tool use and multi-agent collaboration — to manage retrieval strategies dynamically, refine their understanding iteratively and adapt the workflow, in structures that range from sequential steps to adaptive collaboration. We use the term in exactly that sense and no narrower. The rest of this piece is our own engineering analysis of what that changes, and it is an architecture opinion, not a description of any deployment.
Classic RAG in One Paragraph
Retrieval-augmented generation was introduced by Lewis et al. in 2020 (arXiv 2005.11401) as a way to combine a pre-trained language model, which holds knowledge in its parameters, with an explicit non-parametric memory, a dense vector index of Wikipedia reached by a neural retriever. The common enterprise shape built on that idea is simpler than the paper's training recipe: embed the user's question, fetch the top-k most similar chunks from a vector store, place them in the prompt and generate an answer. We will call that "classic RAG". The defining property is not the vector store. It is that the sequence is fixed: retrieval happens exactly once, on every request, and nothing downstream checks whether it worked.
That fixedness is a strength. A fixed pipeline has one retrieval call, one generation call, a predictable latency and cost profile, and a failure you can locate. For a well-scoped corpus and a question type you understand, it is often the right answer. Our enterprise RAG pipeline architecture guide covers the five engineering layers of that design.
What a Fixed Pipeline Gets Wrong
Three weaknesses recur, and each has been named in the research literature.
It retrieves whether or not retrieval is needed. The Self-RAG paper (Asai et al., arXiv 2310.11511) observes that indiscriminately retrieving a fixed number of passages, regardless of whether retrieval is necessary or the passages are relevant, can reduce a language model's versatility and lead to unhelpful responses. A greeting, a follow-up that the conversation already answers, or a question the model can handle without documents all pay the retrieval cost and risk irrelevant context.
It does not check what it retrieved. The Corrective RAG paper (Yan et al., arXiv 2401.15884) starts from the point that RAG depends heavily on the relevance of the retrieved documents, which raises the question of what the system does when retrieval goes wrong. Its answer is a lightweight retrieval evaluator that scores the retrieved documents and triggers different actions depending on confidence. A classic pipeline has no such step: if the top-k are poor, the model is still asked to answer from them.
It gets one shot at a question that may need several. This one is our own observation rather than a finding from those papers. A question that spans two systems, needs a comparison across documents, or must first be decomposed cannot be answered well by a single similarity search on the original wording. When we describe hybrid retrieval over knowledge graphs, part of the motivation is the same limit.
What an Agent Adds to the Pipeline
The four headings follow the design patterns named in the survey's definition. The descriptions are our reading of them.
Planning
Before retrieving, the agent decides what to ask. A compound question is split into sub-questions, each retrieved separately and combined. A vague question can be rewritten into one the index can match. Planning is what turns "one search on the user's words" into a sequence.
Tool use and routing
The agent chooses where to look: a vector index, a keyword or SQL source, a graph, an internal API, or nowhere because the answer does not need documents. In a classic pipeline that choice was made once by the architect. In an agentic one it is made per request, which is also where tool-calling reliability starts to matter.
Reflection
After retrieving, the agent grades the evidence. Is it relevant, sufficient, contradictory? If not, it rewrites the query, tries another source or stops and says it cannot answer. Self-RAG and Corrective RAG are research examples of this evaluation step.
Multi-agent collaboration
For larger systems the work can be divided among specialised agents, for example one per data source and one to synthesise, coordinated by a supervisor. This is the most powerful pattern and the most expensive to operate. Do not start here.
Classic RAG vs Agentic RAG: Our Engineering Comparison
| Dimension | Classic RAG | Agentic RAG |
|---|---|---|
| When retrieval happens | Always, once per request | Decided per request; can be skipped or repeated |
| Query handling | Original wording embedded as is | Can be decomposed, rewritten or routed |
| Sources | One configured index | Several, chosen at run time |
| Quality check on retrieved context | None built in | Explicit grading step, with retry or abstain |
| Number of model calls | Typically one generation call | Several: planning, grading and generation |
| Latency and cost | Predictable and low | Variable and higher; bounded only by the step limit you set |
| Execution path | Same every time | Differs by request, so harder to reproduce |
| Typical failure | Poor retrieval, confidently answered | Loops, wasted steps, wrong routing, injected instructions |
| Evaluation | Retrieval and answer quality | All of that, plus the path taken |
A Bounded Agentic Retrieval Loop
MAX_STEPS = 3 # a hard budget; without one, an agentic loop has no cost ceiling
def answer(question: str, tenant_ctx) -> Answer:
plan = plan_retrieval(question) # may say: "no retrieval needed"
if not plan.needs_retrieval:
return generate(question, context=[])
evidence: list[Chunk] = []
query = plan.first_query
for step in range(MAX_STEPS):
chunks = retrieve(plan.source, query, tenant_ctx) # tenant scope applied in code
verdict = grade(question, chunks) # relevant? sufficient? contradictory?
evidence.extend(c for c in chunks if verdict.keeps(c))
if verdict.sufficient:
break
query = rewrite(question, evidence, verdict.gaps) # try again with a better query
if not evidence:
return abstain("I could not find a supported answer in the available sources.")
return generate(question, context=evidence, cite=True)Illustrative and vendor-neutral. plan, retrieve, grade, rewrite and generate stand for your own components; this is not any framework's API. The point is the structure: the loop is explicit and has a hard step budget.
Agentic RAG buys flexibility with cost and unpredictability. Every extra planning or grading step is another model call and more latency. Because the path differs by request, a failure is harder to reproduce, and evaluating the system means evaluating the path as well as the answer. The survey itself lists evaluation, coordination, memory management, efficiency and governance as open challenges. There is also a security consequence: an agent that reads retrieved text and then decides its next action has given untrusted content a route to influence its behaviour. Treat retrieved documents as untrusted input and read our prompt injection defence guide before giving a retrieval agent any tool that can write.
When to Stay With Classic RAG
Our default advice is to start with classic RAG and move only where evidence says to. If your questions are mostly answered from one well-indexed corpus, if latency and cost must be predictable, or if a single failure path must be easy to audit, a fixed pipeline with good chunking, hybrid retrieval and re-ranking will usually be the better engineering choice. Adding an agent to a retrieval problem that is actually a data-quality problem produces a more expensive system with the same wrong answers.
The signal to move is specific. Run your continuous RAG evaluation first. If the failures cluster into questions that need decomposition, questions that need a source the pipeline does not query, or answers given from irrelevant context, then planning, routing or a grading step addresses a measured problem. Add one pattern at a time, and keep the step budget.
A Decision Checklist
- Have you measured where the current pipeline fails, by question type, before adding an agent?
- Do the failures need decomposition, source routing or evidence grading? If they are chunking or data-quality failures, fix those first.
- Is there a hard step limit and an explicit "abstain" outcome, so the loop cannot run unbounded or invent an answer?
- Is the tenant, user and permission scope applied in code on every retrieval call, not left to the agent's judgement?
- Are retrieved documents treated as untrusted input, and does the agent have the minimum tools it needs?
- Can you replay a request and see the path it took (plan, queries, grades), so failures are diagnosable?
- Have you compared cost and latency per request with the fixed pipeline on the same test set?
How Inductivee Intelligence Approaches This
This section is a design position, not a description of a client engagement. We do not publish client case studies; see our transparency page. When we scope a retrieval system for a Cognitive Web Portal, we begin with the fixed pipeline and an evaluation set, and we add planning, routing or grading only against failure categories the evaluation shows. Where we add an agent, we give it a step budget, an abstain path, and retrieval tools that enforce access control in code rather than in the prompt.
If you are deciding whether a retrieval problem needs an agent at all, an AI readiness assessment is a reasonable place to start, because the answer usually depends on the data and the question mix more than on the framework.
Sources (as of 16 November 2026)
Each was opened on 16 November 2026. The papers are preprints; we cite what their abstracts and pages state.
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv 2005.11401 (submitted 22 May 2020)
- Singh et al., Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG, arXiv 2501.09136 (submitted 15 January 2025; latest version dated 1 April 2026 on arXiv)
- Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, arXiv 2310.11511 (17 October 2023)
- Yan et al., Corrective Retrieval Augmented Generation, arXiv 2401.15884 (submitted 29 January 2024; revised 7 October 2024)
Frequently Asked Questions
What is agentic RAG?
Is agentic RAG better than classic RAG?
Do Self-RAG and Corrective RAG count as agentic RAG?
How do you stop an agentic RAG system running up cost?
Does agentic RAG change the security model?
Written By
Inductivee Intelligence Team
AuthorAgentic AI Engineering Team
The Inductivee Intelligence engineering team — multi-agent orchestration, RAG pipeline architecture, and data liquidity engineering. Our writing is grounded in what we actually build, break, and operate in production.
Inductivee Intelligence is a remote-first agentic AI engineering studio. Our engineering content is written by active practitioners and technically reviewed before publication.
Engineer This With Inductivee Intelligence
The engineering patterns in this article are what our team builds into production every day. Explore the related service to see how we deliver this capability at enterprise scale.
Cognitive Web Portals
Enterprise RAG portals and natural-language gateways — we turn your enterprise data into an interactive, self-service AI assistant grounded in your own knowledge.
ServiceCognitive Data Platforms
Cognitive data platforms and generative BI engineering — we transform raw enterprise data into a reasoning knowledge base for LLMs and autonomous agents. Built on vector databases, semantic ETL, and conversational analytics.
Related Articles
RAG Pipeline Architecture for the Enterprise: Five Layers Beyond the Basic Chatbot
RAG Evaluation in Production: Building Continuous Quality Monitoring
Knowledge Graph RAG: Hybrid Architecture for Complex Enterprise Reasoning
Tool-Calling Architecture: Designing Reliable Function Execution for AI Agents
Ready to Build This Into Your Enterprise?
Inductivee Intelligence engineers agentic systems, RAG pipelines, and enterprise data liquidity solutions. Let's scope your project.
Start a Project