Context Engineering: The Discipline That Replaced Prompt Engineering

Why the Name Changed

Prompt engineering, taken at face value, is about writing better prompts. That framing captured what mattered when the input to the model was one string — a user question, maybe a system message, and that was it. In 2023 it was the right framing. In 2026 it is a partial one.

A modern production LLM call has a system prompt, an accumulated conversation history, retrieved documents from RAG, tool outputs from prior turns, agent scratchpad reasoning, examples pulled from a few-shot store, structured data injected from your application, and maybe compressed memory summarising prior sessions. All of this is 'the context.' The prompt — the actual string you wrote — is one component. Often not the most consequential one.

Engineers running production LLM systems noticed that most of their failures were not prompt failures. They were context failures: the retrieval brought back the wrong document, the memory got stale, the tool output was 400 tokens of JSON when 40 tokens of summary was what the model needed, the system prompt was so long it dominated the model's attention. Fixing the prompt did nothing. Fixing what surrounded the prompt fixed everything.

The term 'context engineering' — popularised by Andrej Karpathy and taken up widely through 2025 — put a name on that reality. It is not a new discipline invented from scratch; it is the recognition that everything a working LLM team was already doing was better described as engineering the context than engineering the prompt.

What Actually Lives in the Context Window

It helps to enumerate. For a modern production agent, the context that reaches the model on any given turn typically contains, in order of appearance:

System instructions. The stable rules — what the assistant is, what it may not do, output format constraints, tool descriptions. This layer changes rarely. Its cost is amortised via prompt caching if you have wired it correctly.

Task-level guidance. For agents with distinct modes or roles, the mode-specific instructions — a triage agent's routing rubric, a specialist agent's domain guidance. Often loaded conditionally.

Retrieved knowledge. RAG payload. The chunks the retriever thinks are relevant to the current query. The single largest variable, both in token budget and in quality impact.

Working memory. A running summary of prior conversation turns, or of prior sessions for a long-lived agent. Compact by construction — a hundred to a few hundred tokens summarising a longer history.

Tool descriptions and prior tool outputs. For agents in the middle of a loop, the descriptions of available tools and the results of any tool calls made this turn or in recent turns. Grows quickly if not managed.

Few-shot exemplars. For tasks where you have curated golden input-output pairs, a small number of them injected as demonstrations. Effective, expensive if uncalibrated.

The immediate user turn. The actual current input.

Scratchpad / reasoning traces. For chain-of-thought or ReAct-style agents, the model's own working notes from this turn.

On a modern model with a 200k+ token window, all of this can technically fit. In practice, blindly filling the window is one of the most reliable ways to make model quality worse — attention is expensive, cost is linear, and the model's ability to find the relevant token in a very long context degrades measurably. The engineering discipline is deciding what belongs, in what order, at what fidelity, for this particular turn.

The Six Techniques That Make Up Context Engineering

The Evaluation Loop for Context Quality

Context engineering is not something you get right by intuition. It is something you get right by measurement, and specifically by measurement that separates retrieval quality from generation quality from context-structuring quality. A single 'accuracy' number is not enough because it does not tell you where to invest next.

The evaluation stack that works has three layers. Retrieval evaluation — recall@k, MRR, NDCG on a golden set of query-document pairs — tells you whether the retriever finds the right context. Wrong here means fix the retriever, not the prompt. Groundedness evaluation — for RAG, is every claim in the answer supported by the retrieved context — tells you whether the model is using the context it was given. Wrong here often means fix the prompt or the structuring, not the retrieval. Task quality — end-to-end, is the answer correct — tells you whether the whole thing is working, but only after you have separated the previous two.

The frameworks that support this well are Ragas, DeepEval, Inspect, and Langfuse's built-in evaluators. Wire them into your CI so a context regression fails the build before it fails a user.

A Concrete Context Assembly Pattern

# context_assembler.py — explicit budget, cached prefix, structured layout

from dataclasses import dataclass

@dataclass
class ContextBudget:
    system: int = 500
    tools: int = 500
    memory: int = 500
    retrieval: int = 2000
    scratchpad: int = 300
    user_turn: int = 500
    headroom: int = 200

    @property
    def total(self) -> int:
        return sum([self.system, self.tools, self.memory, self.retrieval,
                    self.scratchpad, self.user_turn, self.headroom])


def assemble_context(
    *,
    system_prompt: str,        # STABLE — top of context, cached
    tool_descriptions: str,     # STABLE — cached
    session_memory: str,        # slowly-changing summary
    retrieved_chunks: list[str],
    user_turn: str,
    scratchpad: str,
    budget: ContextBudget = ContextBudget(),
) -> list[dict]:
    """Assemble the messages payload with explicit budget enforcement.

    Layout for cache-friendliness:
      1. System instructions (stable, cached)
      2. Tool descriptions (stable, cached)
      3. Session memory (slowly-changing)
      4. Retrieval context (per-turn)
      5. Scratchpad (per-turn)
      6. User message (per-turn)
    """
    # ---- Enforce budgets (truncation preferred over silent overflow) ----
    system_prompt = truncate_tokens(system_prompt, budget.system)
    tool_descriptions = truncate_tokens(tool_descriptions, budget.tools)
    session_memory = summarise_to_budget(session_memory, budget.memory)
    retrieved_chunks = rank_and_pack_to_budget(retrieved_chunks, budget.retrieval)
    scratchpad = truncate_tokens(scratchpad, budget.scratchpad)
    user_turn = truncate_tokens(user_turn, budget.user_turn)

    # ---- Anthropic cache_control: stable prefix cached for 90% cost reduction ----
    system_blocks = [
        {
            'type': 'text', 'text': system_prompt,
            'cache_control': {'type': 'ephemeral'},
        },
        {
            'type': 'text', 'text': f'\n\n## Available tools\n{tool_descriptions}',
            'cache_control': {'type': 'ephemeral'},
        },
    ]

    # ---- User turn assembled fresh each call ----
    user_content = f'''## Session context
{session_memory}

## Retrieved knowledge
{format_chunks(retrieved_chunks)}

## Scratchpad
{scratchpad}

## User
{user_turn}'''

    return [
        {'role': 'system', 'content': system_blocks},
        {'role': 'user', 'content': user_content},
    ]

Common Anti-Patterns We See

In every enterprise engagement we join partway through, we see the same recurring context anti-patterns.

The first is the unbounded system prompt. Every incident produces a new sentence tacked onto the system prompt until it is 4,000 tokens of accreted rules the model no longer follows. Fix: version the system prompt, move behavioural rules into evaluations that catch violations, keep the prompt tight.

The second is retrieval that returns raw source documents. The retriever returns entire PDF pages, or entire wiki articles, or long tool outputs verbatim. Fix: chunk aggressively, rerank ruthlessly, project down to the specific fields the agent will act on.

The third is memory that never forgets. Every prior turn is included in every subsequent turn, and by turn twenty the model is drowning in its own noise. Fix: hierarchical summarisation, or explicit episodic store retrieval.

The fourth is cache-unfriendly ordering. Content that changes on every call is placed at the top of the context, invalidating the cache and multiplying model costs. Fix: stable prefix, variable suffix, always.

The fifth is treating token spend as free. Context grows call after call, no one is watching, and one day the model bill triples. Fix: budgets in code, alerts on p95 context size, dashboards owned by someone.

How Context Engineering Interacts with the Rest of the Stack

Context engineering is not a standalone discipline. It shows up as a specific concern at every layer of a production LLM system, and organising around it changes how those layers are built.

At the retrieval layer it means treating retrieval quality as a first-class KPI, not an afterthought. Retrieval that returns the wrong thing is a context-engineering failure with a downstream cost.

At the agent framework layer it means picking a framework whose primitives make context assembly explicit. Pydantic AI and DSPy both expose the assembled context; frameworks that hide it behind heavy abstractions make context engineering harder.

At the gateway layer — LiteLLM or equivalent — it means using prompt-caching-aware routing and normalising the shape of the request so caching actually works.

At the observability layer — Langfuse or LangSmith — it means capturing the full assembled context, not just the user turn, on every trace. The number of teams whose observability captures only the input and output and cannot debug a bad answer because they cannot see what the model actually saw is unfortunately large.

At the governance layer it means treating what goes into the context as a policy decision. PII in retrieval chunks, secrets in tool outputs, confidential data in memory — these are all context-engineering concerns as much as they are security ones.

How to Organise Around Context Engineering

Seven Decisions Worth Making Before Your Next Deploy

• Explicit context budget. Write down the tokens each section gets. Enforce in code. Alert when p95 drifts.
• Cache-friendly layout. Stable prefix at the top (system, tools, exemplars), variable content at the bottom (retrieval, scratchpad, user turn). Wire up prompt caching on the provider that supports it.
• Retrieval is a service. Own it separately. Evaluate it separately. Not every agent needs the same retrieval config, but every retrieval config needs its own quality gate.
• Design the forgetting. Decide the memory strategy up front — hierarchical summary, episodic store, sliding window. Unbounded history is a bug.
• Compress tool outputs. Wrap noisy tools in a summarisation or projection layer. Do not let 2,000 tokens of JSON pollute every subsequent turn.
• Trace the full context. Your observability platform must capture what the model actually saw, not just what the user typed. Anything less is debugging blind.
• Layered evaluation. Retrieval quality, groundedness, task success — three separate metrics on three separate dashboards. Treat single-number scoring as a smell.

How To Introduce Context Engineering Into an Existing System

The retrofit that works, in order. First, instrument: capture assembled context on every trace, add per-section token counts, wire retrieval metrics if you have RAG. Second, audit: read a hundred real traces and note the anti-patterns you actually see — the unbounded system prompt, the raw-JSON tool outputs, the memory that never forgets. Third, budget: assign a target token count per section, enforce in code, alert on drift. Fourth, cache: reorder the context so the stable prefix is genuinely stable, wire up the provider's prompt caching. Fifth, evaluate: separate retrieval, groundedness, and task-success metrics; wire them into CI. Sixth, iterate: each fix on this list, in this order, produces measurable improvement. Skipping to step five without doing one through four rarely does.

The discipline is not glamorous — it is not a new model release, it is not a new framework. It is the boring work of making sure the context that reaches the model is the smallest, most relevant, most cache-friendly, most measured version of what the model needs. That work is what separates LLM prototypes from LLM systems.

If you would like help — auditing an existing agent's context and identifying the highest-leverage fixes, designing the retrieval or memory layers for a new system, wiring evaluation metrics into your existing observability platform, or picking between Pydantic AI and DSPy for a workload that needs disciplined context engineering — talk to us. We have run this playbook across dozens of enterprise LLM deployments, and the diagnostic almost always finds the same handful of high-leverage fixes.