DSPy in Production: Programming, Not Prompting, the Enterprise LLM Stack

The Problem DSPy Is Solving

Every serious LLM application team hits the same wall. The first version of a prompt is easy — a few hundred words that mostly work. Six months later the same prompt is two thousand words of accreted edge-case handling, no one is confident that removing any line is safe, and swapping to a new model requires rewriting most of it because the accretion was model-specific. Prompt engineering, done at scale, is the polar opposite of software engineering: it produces fragile artefacts whose behaviour is entangled with their exact wording.

DSPy's premise is that this is not a prompt problem, it is a *methodology* problem. Programming solved this in the 1950s: you write the intent in a high-level language, and a compiler translates it to the details. DSPy applies that model to LLM applications. You describe the task as a signature — 'take these typed inputs, produce these typed outputs, subject to these instructions.' You compose modules that describe *what* should happen: predict, chain-of-thought, retrieve-then-generate, ReAct with tools. A compiler then does what the compiler is supposed to do: turn your intent into the actual prompt strings, and — critically — improve them as you feed it more data.

The payoff is that the source of truth stops being a growing prompt string and starts being a Python program plus a dataset. The compiled prompts are artefacts, versioned like any other build output.

What DSPy Actually Is

At the architecture level DSPy is three connected ideas.

Signatures. A signature declares the input and output types of a step, plus a short instruction. It is written in Python either as a docstring string or as a class with typed fields. The signature is *what* the step does; the prompt that instantiates it is *how* — and DSPy owns the how.

Modules. A module is a composable building block that implements a pattern using one or more signatures. dspy.Predict runs a signature once. dspy.ChainOfThought runs it with a reasoning field. dspy.ReAct implements the plan-act-observe loop with tools. dspy.Retrieve pulls from a retriever. Modules compose — you can build a RAG pipeline as Retrieve → ChainOfThought in a handful of lines.

Optimisers (also called compilers). This is the piece that makes DSPy different from a nicer prompt library. Given your program, a metric, and a small training set, an optimiser like BootstrapFewShot, MIPROv2, or BootstrapFinetune searches the space of prompt wordings, few-shot exemplars, and even fine-tuned weight updates for a program that scores better on your metric. What you check into version control is your Python module and a compiled artefact — the optimised program is a saved-and-loadable object.

On top of these you get typed structured outputs (DSPy defers to Pydantic when types are used), a caching layer that keeps development cheap, provider-agnostic model access, and integrations with the observability platforms most teams already run (Langfuse, MLflow, Weights & Biases).

How DSPy Compares to the Other Serious Options

Where DSPy Fits in an Enterprise Stack

The mental model that works is: DSPy is where your LLM logic lives, but nothing else. Everything else — ingress, gateway, retrieval, observability, governance — sits at its usual layer and DSPy calls into or out of them.

At the front, a normal FastAPI or Django service handles auth, tenancy, and rate limits. Requests come in as validated Pydantic models and are handed to a DSPy program. The DSPy program owns the LLM decision — one predict, or a compose of predict + chain-of-thought + retrieve + tool calls.

Model calls route through an AI gateway rather than direct provider endpoints. DSPy configures its LM as an OpenAI-compatible endpoint by default, so pointing at LiteLLM (or Portkey, or your own proxy) is one line of config. Budgets, keys, and provider routing then live at the gateway, not in every DSPy module.

Retrieval — for RAG-shaped programs — sits behind a normal retriever interface. DSPy has adapters for the common vector databases, or you can wrap your own. The retriever is not owned by DSPy; the DSPy program uses it, in the same way it uses tools.

Compiled DSPy programs are artefacts. You produce them in a build step (offline against your training set), version them alongside model versions in your artefact store, and load the appropriate one at runtime for the deployment target. This is what turns 'prompt engineering' from a manual craft into a reproducible build.

A Production-Shaped DSPy Program

# support_triage.py — DSPy program with typed signatures

import dspy
from pydantic import BaseModel, Field

# ---- Configure model access via the AI gateway ----
dspy.settings.configure(
    lm=dspy.LM(
        model='openai/claude-sonnet-4-5',       # routed by the LiteLLM proxy
        api_base='https://gateway.internal.example.com/v1',
        api_key=os.environ['GATEWAY_KEY'],
    ),
    rm=CompanyRetriever(),                       # our own retriever adapter
)

# ---- Typed outputs — Pydantic model, DSPy validates against it ----
class SupportDecision(BaseModel):
    action: str = Field(pattern=r'^(answer|route_billing|route_returns|escalate)$')
    rationale: str
    citations: list[str] = Field(default_factory=list)
    confidence: float = Field(ge=0.0, le=1.0)

# ---- Signature: the intent, not the prompt ----
class TriageSignature(dspy.Signature):
    """Given a support ticket and retrieved policy snippets, decide the next step.
    Cite the policies you relied on. Escalate anything you are not confident about."""
    ticket_text: str = dspy.InputField()
    retrieved_policies: list[str] = dspy.InputField()
    decision: SupportDecision = dspy.OutputField()

# ---- Module: compose retrieval + chain-of-thought reasoning ----
class SupportTriage(dspy.Module):
    def __init__(self, num_policies: int = 4):
        super().__init__()
        self.retrieve = dspy.Retrieve(k=num_policies)
        self.decide = dspy.ChainOfThought(TriageSignature)

    def forward(self, ticket_text: str) -> SupportDecision:
        policies = self.retrieve(ticket_text).passages
        result = self.decide(ticket_text=ticket_text, retrieved_policies=policies)
        return result.decision

# ---- Build step: compile against your labelled training set ----
def compile_and_save():
    from dspy.teleprompt import MIPROv2

    trainset = load_labelled_tickets()   # your golden data
    metric = ticket_decision_score        # your metric function

    program = SupportTriage()
    optimiser = MIPROv2(metric=metric, auto='medium')
    compiled = optimiser.compile(program, trainset=trainset)

    compiled.save('artifacts/support_triage.v3.json')

# ---- Runtime ----
def handle_ticket(ticket_text: str) -> SupportDecision:
    program = SupportTriage()
    program.load('artifacts/support_triage.v3.json')
    return program(ticket_text=ticket_text)

Optimisation Is a Build Step, Not a Runtime Event

One of the easiest mistakes new DSPy teams make is treating compilation as something that happens at runtime. It should not. Compilation is a build step — expensive, reproducible, versionable, done offline against a training set and a metric.

The pipeline that works: a CI job runs when your training data or your program code changes. It recompiles the DSPy modules against the current dataset, produces a versioned artefact, runs a validation pass against a held-out set, and only promotes the artefact if metrics have not regressed. Your runtime service loads a specific compiled artefact by version, exactly like it loads a specific model version. Rolling back a bad prompt change is 'change the artefact version in the deploy config,' not 'find the string and undo it.'

The corollary is that your training set — even a small one — is now a first-class artefact. Twenty to a hundred labelled examples per module is enough to see real gains from BootstrapFewShot. Larger sets (several hundred to a few thousand) unlock the more expensive optimisers and, if you want, weight-level fine-tuning through BootstrapFinetune. The number matters less than the discipline: labelled examples that reflect the distribution you actually see in production.

Model Swaps as a Recompile, Not a Rewrite

The most valuable operational property DSPy gives you shows up when a new model comes out. In the hand-tuned prompt world, a new model means a week of prompt-string re-engineering to recover the quality you had. In DSPy, a new model means changing the lm= config, running the compile step against your existing dataset, and reviewing the metric report. If the new model is better, the compiled artefact is better. If it is worse, the metric tells you before you deploy.

This is why teams running heterogeneous stacks — Claude for reasoning-heavy modules, a smaller open-weights model for classification, a cheap fast model for a routing step — gravitate to DSPy over hand-tuned prompts. The compile step normalises the differences between models into a single build artefact per module. What used to be 'we cannot swap models because the prompt is entangled with the model's quirks' becomes 'the compile step handles the quirks.'

Where DSPy Tends To Be the Right Choice

Where It Is Not the Right Choice

If your team is not Python-native, DSPy is Python-first and there is no equivalent in other ecosystems yet. If your product runs on free-form conversational output that resists metric definition — a chatbot whose success is measured by user vibes — DSPy's optimiser has nothing to optimise for, and you are effectively using it as a slightly-nicer prompt library. If your workflow is orchestration-heavy (branching, parallelism, human-in-the-loop pauses), you want LangGraph or an agent runtime around the DSPy modules, not DSPy itself doing the orchestration. And if you have zero training data and no path to any, DSPy's core value proposition — optimise against data — does not apply, and hand-tuned prompts remain your baseline.

Seven Decisions Worth Making Before Your First Deploy

• Signature first, prompt never. Write the signature — inputs, outputs, one-line docstring. Do not write the prompt. If you catch yourself writing prompt text, you are working against the framework.
• Metric first, then optimiser. Pick the metric that reflects 'the answer was correct enough' *before* you compile. If you cannot write a metric function, you cannot use the optimiser productively.
• Small dataset, real distribution. Twenty to a hundred labelled examples that reflect your real production distribution beat a thousand that do not. Curate over collect.
• Compile in CI, load at runtime. Never compile at request time. Compile as a build step, version the artefact, load a specific version at runtime.
• Optimiser progression. Start with BootstrapFewShot. Move to MIPROv2 when you have a stable metric and want more gain. Reach for BootstrapFinetune only when prompt-level optimisation has run out and you can justify the weight-tuning cost.
• Route through a gateway. Configure DSPy's LM as your AI gateway endpoint, never a direct provider. Keys, budgets, and provider swaps then live at one layer.
• Wire observability from day one. DSPy has native integrations for Langfuse, MLflow, and Weights & Biases. Enabling them costs three lines and gives you queryable traces of every module invocation, including the compiled prompt that ran.

How To Approach a DSPy Pilot

A pilot that produces a real answer looks like this. Pick one narrow LLM decision your system makes today with a hand-tuned prompt — a classifier, a triage, a structured extractor. Collect fifty to a hundred labelled examples from real traffic and set aside twenty for validation. Write the signature and the module. Define a metric function. Compile with BootstrapFewShot first, then MIPROv2. Compare the compiled artefact against your existing hand-tuned prompt on the validation set. Only then decide whether to switch.

By the end you have a concrete answer to three questions: is the compiled version measurably better on our metric, how much did the compile cost, and how much prompt-engineering effort would we have needed to reach the same place. If the compiled version wins and the effort ratio is favourable, roll it into the artefact pipeline and start applying the same discipline to the next module.

If you would like help with the architecture — designing signatures and metrics, integrating with your gateway and observability platform, building the compile-and-promote CI pipeline, or deciding between DSPy and a Pydantic-AI + hand-tuned prompt baseline — talk to us. We have deployed DSPy for classification, extraction, RAG-with-reasoning, and multi-step tool-using agents across regulated and non-regulated enterprises, and the reference architecture is repeatable in ways most first attempts are not.