Langfuse in Production: Enterprise LLM Observability Architecture for Multi-Model AI Systems

Why Standard Observability Falls Short for LLM Systems

A traditional web request is easy to characterise. You have a status code, a latency, a response size, a user, an endpoint. The behaviour is deterministic enough that classical APM tools can answer the question 'is this working' with high confidence.

An LLM request is a very different shape. A single user query might expand into a chain: a retrieval step, a rerank step, three or four model calls with different prompts, a tool invocation, another model call to summarise. Each of those steps has its own prompt version, its own model, its own token count, its own cost, and its own correctness question. The final response can be well-formed HTTP-200 and still be wrong.

On top of that, the same prompt against the same model on the same input can produce different outputs on different days, quietly regress after a model update, or start refusing tasks that used to work. Classical APM does not surface any of that.

What you need is a layer that captures the full call graph, the exact prompt content and version, the model and parameters, the token usage and cost, and — critically — some way to evaluate whether the output was any good. That is the space Langfuse and its peers occupy.

What Langfuse Actually Is

Langfuse is an open-source LLM engineering platform, MIT-licensed, maintained as a company-backed open-source project. At the architecture level it is four connected capabilities behind one dashboard and one API.

Tracing. Every LLM interaction — a full multi-step chain, not just a single call — is captured as a trace made up of spans. Prompts, completions, model, parameters, latency, token counts, and computed cost are attached to each span. SDKs for Python and JavaScript, plus native integrations with the major frameworks and OpenTelemetry compatibility, mean instrumentation is usually a one-line change per call site.

Prompt management. Prompts live in Langfuse rather than in code strings. Each prompt is versioned, has labels (production, staging, experimental), and is fetched by the application at runtime. Rolling back a prompt does not require a deploy.

Evaluations. Traces can be scored — by users, by rules, by other LLMs (LLM-as-judge), or by human reviewers — against a dataset. This is how you turn 'the answer felt off' into a measurable regression.

Datasets and experiments. Curated input/output pairs support offline evaluation runs against new prompts or models before anything ships.

What Langfuse deliberately is not: it is not a gateway, not a router, not a guardrails engine. It sits behind those layers and records what happened. That separation is the point — it lets you swap gateways or models without losing observability continuity.

How Langfuse Compares to the Other Serious Options

Deployment Topology

Langfuse offers a managed cloud (US and EU regions) with a free tier for teams getting started, and a self-hosted deployment that is fully open source under MIT. For enterprises the decision usually comes down to data-residency rules, security posture, and how tightly you want observability coupled to the rest of your platform.

Self-hosted Langfuse is a containerised web/API server, a Postgres database for metadata and configuration, and — since v3 of the platform — ClickHouse for high-volume trace storage. Redis is used for queuing and caching, and object storage (S3, GCS, or MinIO) holds larger event blobs. This is a heavier footprint than the earlier all-Postgres deployment but it is what allows the platform to handle millions of traces without the metadata database becoming a bottleneck.

A Production-Shaped Self-Hosted Deployment

# docker-compose.production.yml (sketch — real deployments use Helm on k8s)

services:
  langfuse-web:
    image: langfuse/langfuse:3
    depends_on: [postgres, clickhouse, redis, minio]
    environment:
      # Core
      DATABASE_URL: postgresql://langfuse:${PG_PASSWORD}@postgres:5432/langfuse
      NEXTAUTH_URL: https://langfuse.internal.example.com
      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET}
      SALT: ${SALT}
      ENCRYPTION_KEY: ${ENCRYPTION_KEY}

      # High-volume trace store
      CLICKHOUSE_URL: http://clickhouse:8123
      CLICKHOUSE_USER: langfuse
      CLICKHOUSE_PASSWORD: ${CH_PASSWORD}

      # Async ingestion
      REDIS_HOST: redis
      REDIS_PORT: 6379

      # Blob storage for large events
      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: langfuse-events
      LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: https://s3.internal.example.com

      # Enterprise SSO (SAML / OIDC — attach to Okta, Entra, etc.)
      AUTH_OKTA_CLIENT_ID: ${OKTA_CLIENT_ID}
      AUTH_OKTA_CLIENT_SECRET: ${OKTA_CLIENT_SECRET}
      AUTH_OKTA_ISSUER: https://example.okta.com

      # Sensible defaults for a serious deployment
      LANGFUSE_INIT_ORG_NAME: acme
      TELEMETRY_ENABLED: false

  langfuse-worker:
    # Async ingestion worker — scales horizontally
    image: langfuse/langfuse-worker:3
    depends_on: [postgres, clickhouse, redis]
    environment: *worker-env
    deploy:
      replicas: 3

  postgres:
    image: postgres:16
    # Managed RDS / Cloud SQL in reality

  clickhouse:
    image: clickhouse/clickhouse-server:24
    # Managed ClickHouse Cloud or self-hosted cluster in reality

  redis:
    image: redis:7

  minio:
    image: minio/minio
    # Or use S3 / GCS directly

Instrumentation Patterns That Age Well

The biggest observability mistake we see is instrumenting only the final LLM call. You end up with rows of completions and no way to reconstruct why one was slow or wrong. The pattern that ages well is to open a trace at the top of the user-visible operation, and to record every non-trivial step as a span underneath it — retrievals, reranks, guardrail checks, tool calls, sub-agent invocations, and the LLM calls themselves.

A second pattern worth adopting early: attach a stable session ID and user ID to every trace. Sessions let you reconstruct multi-turn conversations. User IDs let you slice cost, latency, and quality by tenant, by cohort, or by the one user who is complaining. Retrofitting these later is painful.

A third: fetch prompts from Langfuse rather than hardcoding them. The runtime cost is negligible — prompts cache locally — and the operational upside is large: a bad prompt can be rolled back without a deploy, and every trace records which exact version was used, which is what makes regression analysis possible at all.

Evaluations: The Part Most Teams Skip

Tracing tells you what happened. Evaluations tell you whether it was any good. A production LLM system without evaluations is a system where quality drifts silently and the first sign of trouble is a customer complaint.

Langfuse supports several evaluation modes and they compose. User feedback — thumbs up/down, ratings — is the cheapest signal and the easiest to wire up. Rule-based scores catch structural issues: JSON that fails to parse, responses that exceed a length threshold, answers that include a forbidden term. LLM-as-judge evaluations use another model to score outputs against a rubric — useful for correctness, tone, faithfulness to retrieved sources. Human review is expensive but produces the ground truth that your automated evaluators are ultimately calibrated against.

The operational pattern that works: pick two or three metrics that matter for your use case (faithfulness for RAG, task-success for agents, refusal-rate for customer-facing bots), wire them up to a rolling window of production traces, and put them on a dashboard that someone actually looks at. Any drop of more than a few points, week over week, is a regression to investigate — usually caused by a model update, a prompt change, or a retrieval regression.

OpenTelemetry, and How Langfuse Fits Alongside Existing Observability

Langfuse is OpenTelemetry-compatible: it accepts OTLP traces from any OTel-instrumented application and it can export to OTel-compatible backends. In practice this means the LLM-specific view lives in Langfuse — traces, prompt versions, evaluations, cost — while the ops-facing view stays in whatever you already use for infrastructure APM.

The cleanest pattern we deploy: application code emits standard OTel spans for HTTP handlers, database calls, queue jobs, and so on, and dual-emits LLM-specific spans through the Langfuse SDK (or via the OTel-native path). Datadog or your APM of choice gets the ops signal. Langfuse gets the LLM signal. There is no fight over which is the source of truth because they are the source of truth for different questions.

Where Langfuse Tends To Be the Right Choice

Seven Decisions Worth Making Deliberately Before Your First Deploy

• Cloud versus self-host. Cloud is faster to prove value; self-host is the answer if data residency, regulator scrutiny, or platform integration matter. Do not pick self-host to save money — the ops cost of ClickHouse and Postgres at scale is real.
• Trace granularity. Decide what is worth capturing as a span up front. Every retrieval, every guardrail, every tool call — yes. Every internal function call — no. Overspending on granularity is expensive and slows dashboards; under-spending leaves you blind to the interesting failures.
• Session and user ID discipline. Attach both to every trace from day one. Retrofitting these is a data-migration project. Doing it up front is a single decorator.
• Prompt lifecycle. Move prompts from string literals into Langfuse-managed prompts before you have thirty of them. Once you can roll back and A/B without a deploy, prompt work stops being scary.
• Redaction policy. Write it down before the first trace lands. What must never be logged (secrets, PII, PHI). What is safe (task IDs, tenant IDs). What is masked (email → hash). Bake it into the SDK wrapper so no application team has to think about it.
• Evaluation cadence. Pick two or three metrics per use case. Wire them to a rolling window. Put them on a dashboard someone owns. Without this, tracing is diagnostics; with it, tracing is quality.
• Retention. Traces accumulate quickly. Decide up front how long you keep full trace data versus aggregated metrics. A common pattern is 30 days full fidelity, 12 months aggregates, indefinite dataset snapshots for regulated workloads.

How To Approach a Langfuse Pilot

A pilot that works looks like this. Pick one product surface — a single agent, a single chatbot, one RAG endpoint — with real traffic. Deploy Langfuse in self-host or cloud, whichever will match production. Instrument that one surface end-to-end: every retrieval, every model call, every tool call, with a session ID and user ID on every trace. Move its prompts into Langfuse-managed prompts. Wire up one or two evaluations — user thumbs plus one LLM-as-judge is a good starting pair. Run this for two weeks.

At the end you will have a concrete answer to four questions: what does a normal week of LLM traffic look like, what is it costing, what is failing, and where do prompts need to move. That answer is what turns Langfuse from a tool your team likes into a platform investment your leadership signs off on.

If you would like help with the architecture — the self-host vs cloud decision, the instrumentation patterns, the evaluation setup, or connecting Langfuse to an existing governance framework or AI gateway — talk to us. We have done this deployment across teams running single agents and teams running a full estate of models behind vLLM, and the patterns are surprisingly consistent.