Skip to main content
RAG & Retrieval

Agentic RAG vs Classic RAG: What Changes in the Architecture

Classic RAG retrieves once, in a fixed pipeline, whether or not retrieval is needed. Agentic RAG puts an agent in charge of planning, routing and checking retrieval. Here is the precise definition, what the agent actually adds, what it costs, and when a fixed pipeline is still the better engineering choice.

Inductivee Intelligence Team· AI EngineeringNovember 8, 202611 min read
TL;DR

"Agentic RAG" is used loosely, so here is the definition this post uses, once. In the survey Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (Singh et al., arXiv 2501.09136, first submitted 15 January 2025), Agentic RAG means embedding autonomous AI agents into the RAG pipeline. Those agents use agentic design patterns — reflection, planning, tool use and multi-agent collaboration — to manage retrieval strategies dynamically, refine their understanding iteratively and adapt the workflow, in structures that range from sequential steps to adaptive collaboration. We use the term in exactly that sense and no narrower. The rest of this piece is our own engineering analysis of what that changes, and it is an architecture opinion, not a description of any deployment.

Classic RAG in One Paragraph

Retrieval-augmented generation was introduced by Lewis et al. in 2020 (arXiv 2005.11401) as a way to combine a pre-trained language model, which holds knowledge in its parameters, with an explicit non-parametric memory, a dense vector index of Wikipedia reached by a neural retriever. The common enterprise shape built on that idea is simpler than the paper's training recipe: embed the user's question, fetch the top-k most similar chunks from a vector store, place them in the prompt and generate an answer. We will call that "classic RAG". The defining property is not the vector store. It is that the sequence is fixed: retrieval happens exactly once, on every request, and nothing downstream checks whether it worked.

That fixedness is a strength. A fixed pipeline has one retrieval call, one generation call, a predictable latency and cost profile, and a failure you can locate. For a well-scoped corpus and a question type you understand, it is often the right answer. Our enterprise RAG pipeline architecture guide covers the five engineering layers of that design.

What a Fixed Pipeline Gets Wrong

Three weaknesses recur, and each has been named in the research literature.

It retrieves whether or not retrieval is needed. The Self-RAG paper (Asai et al., arXiv 2310.11511) observes that indiscriminately retrieving a fixed number of passages, regardless of whether retrieval is necessary or the passages are relevant, can reduce a language model's versatility and lead to unhelpful responses. A greeting, a follow-up that the conversation already answers, or a question the model can handle without documents all pay the retrieval cost and risk irrelevant context.

It does not check what it retrieved. The Corrective RAG paper (Yan et al., arXiv 2401.15884) starts from the point that RAG depends heavily on the relevance of the retrieved documents, which raises the question of what the system does when retrieval goes wrong. Its answer is a lightweight retrieval evaluator that scores the retrieved documents and triggers different actions depending on confidence. A classic pipeline has no such step: if the top-k are poor, the model is still asked to answer from them.

It gets one shot at a question that may need several. This one is our own observation rather than a finding from those papers. A question that spans two systems, needs a comparison across documents, or must first be decomposed cannot be answered well by a single similarity search on the original wording. When we describe hybrid retrieval over knowledge graphs, part of the motivation is the same limit.

What an Agent Adds to the Pipeline

The four headings follow the design patterns named in the survey's definition. The descriptions are our reading of them.

Planning

Before retrieving, the agent decides what to ask. A compound question is split into sub-questions, each retrieved separately and combined. A vague question can be rewritten into one the index can match. Planning is what turns "one search on the user's words" into a sequence.

Tool use and routing

The agent chooses where to look: a vector index, a keyword or SQL source, a graph, an internal API, or nowhere because the answer does not need documents. In a classic pipeline that choice was made once by the architect. In an agentic one it is made per request, which is also where tool-calling reliability starts to matter.

Reflection

After retrieving, the agent grades the evidence. Is it relevant, sufficient, contradictory? If not, it rewrites the query, tries another source or stops and says it cannot answer. Self-RAG and Corrective RAG are research examples of this evaluation step.

Multi-agent collaboration

For larger systems the work can be divided among specialised agents, for example one per data source and one to synthesise, coordinated by a supervisor. This is the most powerful pattern and the most expensive to operate. Do not start here.

Classic RAG vs Agentic RAG: Our Engineering Comparison

DimensionClassic RAGAgentic RAG
When retrieval happensAlways, once per requestDecided per request; can be skipped or repeated
Query handlingOriginal wording embedded as isCan be decomposed, rewritten or routed
SourcesOne configured indexSeveral, chosen at run time
Quality check on retrieved contextNone built inExplicit grading step, with retry or abstain
Number of model callsTypically one generation callSeveral: planning, grading and generation
Latency and costPredictable and lowVariable and higher; bounded only by the step limit you set
Execution pathSame every timeDiffers by request, so harder to reproduce
Typical failurePoor retrieval, confidently answeredLoops, wasted steps, wrong routing, injected instructions
EvaluationRetrieval and answer qualityAll of that, plus the path taken

A Bounded Agentic Retrieval Loop

python
MAX_STEPS = 3  # a hard budget; without one, an agentic loop has no cost ceiling


def answer(question: str, tenant_ctx) -> Answer:
    plan = plan_retrieval(question)            # may say: "no retrieval needed"
    if not plan.needs_retrieval:
        return generate(question, context=[])

    evidence: list[Chunk] = []
    query = plan.first_query
    for step in range(MAX_STEPS):
        chunks = retrieve(plan.source, query, tenant_ctx)   # tenant scope applied in code
        verdict = grade(question, chunks)      # relevant? sufficient? contradictory?
        evidence.extend(c for c in chunks if verdict.keeps(c))
        if verdict.sufficient:
            break
        query = rewrite(question, evidence, verdict.gaps)   # try again with a better query

    if not evidence:
        return abstain("I could not find a supported answer in the available sources.")
    return generate(question, context=evidence, cite=True)

Illustrative and vendor-neutral. plan, retrieve, grade, rewrite and generate stand for your own components; this is not any framework's API. The point is the structure: the loop is explicit and has a hard step budget.

Warning

Agentic RAG buys flexibility with cost and unpredictability. Every extra planning or grading step is another model call and more latency. Because the path differs by request, a failure is harder to reproduce, and evaluating the system means evaluating the path as well as the answer. The survey itself lists evaluation, coordination, memory management, efficiency and governance as open challenges. There is also a security consequence: an agent that reads retrieved text and then decides its next action has given untrusted content a route to influence its behaviour. Treat retrieved documents as untrusted input and read our prompt injection defence guide before giving a retrieval agent any tool that can write.

When to Stay With Classic RAG

Our default advice is to start with classic RAG and move only where evidence says to. If your questions are mostly answered from one well-indexed corpus, if latency and cost must be predictable, or if a single failure path must be easy to audit, a fixed pipeline with good chunking, hybrid retrieval and re-ranking will usually be the better engineering choice. Adding an agent to a retrieval problem that is actually a data-quality problem produces a more expensive system with the same wrong answers.

The signal to move is specific. Run your continuous RAG evaluation first. If the failures cluster into questions that need decomposition, questions that need a source the pipeline does not query, or answers given from irrelevant context, then planning, routing or a grading step addresses a measured problem. Add one pattern at a time, and keep the step budget.

A Decision Checklist

  • Have you measured where the current pipeline fails, by question type, before adding an agent?
  • Do the failures need decomposition, source routing or evidence grading? If they are chunking or data-quality failures, fix those first.
  • Is there a hard step limit and an explicit "abstain" outcome, so the loop cannot run unbounded or invent an answer?
  • Is the tenant, user and permission scope applied in code on every retrieval call, not left to the agent's judgement?
  • Are retrieved documents treated as untrusted input, and does the agent have the minimum tools it needs?
  • Can you replay a request and see the path it took (plan, queries, grades), so failures are diagnosable?
  • Have you compared cost and latency per request with the fixed pipeline on the same test set?

How Inductivee Intelligence Approaches This

This section is a design position, not a description of a client engagement. We do not publish client case studies; see our transparency page. When we scope a retrieval system for a Cognitive Web Portal, we begin with the fixed pipeline and an evaluation set, and we add planning, routing or grading only against failure categories the evaluation shows. Where we add an agent, we give it a step budget, an abstain path, and retrieval tools that enforce access control in code rather than in the prompt.

If you are deciding whether a retrieval problem needs an agent at all, an AI readiness assessment is a reasonable place to start, because the answer usually depends on the data and the question mix more than on the framework.

Frequently Asked Questions

What is agentic RAG?

Agentic RAG embeds autonomous AI agents into a retrieval-augmented generation pipeline. According to the Singh et al. survey, the agents use reflection, planning, tool use and multi-agent collaboration to manage retrieval dynamically, refine their understanding iteratively and adapt the workflow. In practice that means the system decides whether, where and how often to retrieve, and checks what it found, instead of retrieving once on a fixed path.

Is agentic RAG better than classic RAG?

Not in general. It is more flexible, and it is also costlier, slower and harder to evaluate, because each request can take a different path with more model calls. For a well-scoped corpus and predictable questions, a fixed pipeline is often the better choice. Move to agentic patterns where evaluation shows failures that planning, routing or evidence grading would fix.

Do Self-RAG and Corrective RAG count as agentic RAG?

They are research examples of the evaluation-and-correction idea that agentic RAG builds on. Self-RAG trains a model to retrieve on demand and critique its own output. Corrective RAG adds a lightweight evaluator that scores retrieved documents and triggers different retrieval actions. Neither is an agent framework in the sense of a planning loop with tools, so treat them as related techniques, not as the definition.

How do you stop an agentic RAG system running up cost?

Set a hard step budget on the retrieval loop, give the agent an explicit way to abstain, measure cost and latency per request against the fixed pipeline on the same test set, and keep the number of models and tools small. Without a budget, nothing bounds the number of calls per question.

Does agentic RAG change the security model?

Yes. An agent that acts on retrieved text lets untrusted content influence its next step. Treat retrieved documents as untrusted input, enforce tenant and permission scope in code on every retrieval call, and keep write-capable tools out of the retrieval agent unless a human approves their use.

Written By

Inductivee Intelligence Team — AI Engineering at Inductivee Intelligence

Inductivee Intelligence Team

Author

Agentic AI Engineering Team

The Inductivee Intelligence engineering team — multi-agent orchestration, RAG pipeline architecture, and data liquidity engineering. Our writing is grounded in what we actually build, break, and operate in production.

Agentic AI ArchitectureMulti-Agent OrchestrationLangChainLangGraphCrewAIMicrosoft Agent Framework
LinkedIn profile

Inductivee Intelligence is a remote-first agentic AI engineering studio. Our engineering content is written by active practitioners and technically reviewed before publication.

Ready to Build This Into Your Enterprise?

Inductivee Intelligence engineers agentic systems, RAG pipelines, and enterprise data liquidity solutions. Let's scope your project.

Start a Project