vLLM in Production: Enterprise Inference Architecture for Self-Hosted LLMs

What vLLM Actually Is

vLLM is an open-source library for fast LLM inference and serving, originally developed at the UC Berkeley Sky Computing Lab and now maintained by a broad community of contributors. It is released under the Apache 2.0 licence. The headline result from the original PagedAttention paper (Kwon et al., SOSP 2023, *Efficient Memory Management for Large Language Model Serving with PagedAttention*) is that careful KV-cache management plus iteration-level scheduling lets a single GPU serve substantially more concurrent requests than the naïve static-batching baselines that earlier inference servers shipped with. The exact speed-up depends on model size, sequence length, and hardware — the paper has the reference figures and the project's benchmark suite is the right source of truth for what to expect on your specific stack.

Underneath the hood, vLLM is a Python library with a C++/CUDA core that exposes two main surfaces. The first is an `LLM` Python class for offline batched inference — useful for evaluation runs, dataset generation, and one-off jobs. The second is an OpenAI-compatible HTTP server (`vllm.entrypoints.openai.api_server`) that speaks the same `/v1/chat/completions` and `/v1/completions` endpoints as the OpenAI API, which means existing client SDKs, LangChain, LangGraph, and most agent frameworks point at it with a one-line base-URL change. For most enterprise deployments, that compatibility is the practical reason vLLM wins over more bespoke serving stacks — your application code does not have to know it is talking to a self-hosted model.

vLLM supports a broad and growing list of model architectures — Llama, Mistral, Mixtral, Qwen, DeepSeek, Gemma, Phi, and many others — through Hugging Face Transformers-compatible model definitions. The supported-models list on the vLLM documentation is the canonical reference and is worth checking before committing to a model choice.

Why PagedAttention and Continuous Batching Matter

To understand what vLLM is doing differently, it helps to be concrete about where the bottleneck sits in LLM inference. Each token a model generates needs the attention mechanism to look back at every previous token, and the keys and values for those previous tokens are kept in GPU memory in the KV cache. A request that has generated 2,000 tokens is carrying ~2,000 tokens worth of KV cache around with it for every subsequent token it generates. KV-cache memory is, in practice, the constraint that determines how many concurrent requests a GPU can serve.

The pre-vLLM convention was to allocate a contiguous KV-cache buffer per request, sized to the request's maximum possible length. That is wasteful: most requests never reach maximum length, so a large fraction of allocated memory sits unused. Worse, the system has to either preallocate for worst-case (severely limiting concurrency) or dynamically reallocate (causing fragmentation). PagedAttention treats KV-cache memory like OS virtual memory — it is divided into fixed-size blocks, each request holds a list of block pointers, and blocks are allocated only when actually needed. The result is much higher KV-cache utilisation, which translates directly into more concurrent requests on the same GPU.

Continuous batching is the scheduling counterpart. Static batching waits for the slowest request in a batch to finish before starting the next batch, which means the GPU spends a lot of time finishing one long generation for a single request while shorter requests sit in the queue. Continuous batching schedules at the iteration level: as soon as one request in the batch finishes a step, a new request can take its slot. The GPU stays utilised; queue latency drops. Together with PagedAttention, this is what gives vLLM its throughput advantage.

How vLLM Compares to the Other Serious Choices

Deployment Topology: Single GPU to Multi-Node

How you deploy vLLM depends on what fits on what hardware. For small and medium models — Llama 3 8B, Mistral 7B, Qwen 7B, anything in the ~7-13B parameter range at FP16 — a single modern GPU with 40-80 GB of memory is usually sufficient, and the deployment is a single container with the OpenAI-compatible server, fronted by your load balancer of choice. This is by far the most common shape and is what most teams should start with.

For larger models — 70B-class and above at FP16 — the model does not fit on a single GPU and you need tensor parallelism across multiple GPUs on a single node. vLLM supports tensor parallelism out of the box via the `tensor_parallel_size` argument. The constraint is fast inter-GPU interconnect: tensor parallelism communicates intermediate activations on every forward pass, so NVLink-class bandwidth is essentially required. Tensor-parallelising across PCIe-only GPUs works but is noticeably slower than the NVLink path; benchmark on your specific hardware before committing.

For models that exceed a single node — typically 400B-class and above at FP16 — you need pipeline parallelism across nodes in addition to tensor parallelism within a node. vLLM supports this through its distributed runtime, with Ray as the coordination layer. Cross-node deployments are operationally harder: you need reliable RDMA networking between nodes, careful failure-domain design (one node failing should not take down the cluster), and an autoscaler that understands the multi-node unit. Most enterprises should not run multi-node deployments until they have exhausted single-node options through quantisation, smaller model variants, or KV-cache offloading.

A practical pattern across all topologies is to run multiple replicas of the deployment behind a load balancer, scaled horizontally on request rate or GPU utilisation. Each replica is independently capable of serving the full model, which gives you both throughput scaling and zone-level fault tolerance. The Kubernetes-native pattern — one Deployment per model variant, each replica a vLLM pod with GPU resource requests, fronted by a Service and an Ingress — works well and is what most enterprise platform teams converge on.

A Production-Shaped vLLM Server Configuration

# ─── PRODUCTION VLLM SERVE COMMAND ──────────────────────────────────
# Illustrative configuration for a Llama-3.1-70B deployment on a
# single H100 node with 8 GPUs and NVLink. Tune values against your
# own workload — the right numbers depend on hardware, model, and
# request mix.

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --quantization fp8 \
  --served-model-name llama-3.1-70b-prod \
  --api-key "$VLLM_API_KEY" \
  --host 0.0.0.0 --port 8000

# Key flags and what they actually do:
#
#   --tensor-parallel-size 8
#     Shard the model across 8 GPUs on this node. Requires NVLink to
#     perform well. Communication happens on every forward pass.
#
#   --gpu-memory-utilization 0.92
#     Fraction of each GPU's memory vLLM is allowed to claim for
#     model weights + KV cache. The remainder is headroom for CUDA
#     activations, NCCL buffers, and graphics-related state. 0.85-0.92
#     is the safe band; pushing higher risks OOM under load spikes.
#
#   --enable-prefix-caching
#     Reuses KV-cache blocks across requests that share a prefix
#     (system prompts, few-shot examples, common RAG context). Huge
#     win for chat-style and RAG workloads; effectively free.
#
#   --enable-chunked-prefill
#     Splits long prefill steps into chunks that interleave with
#     decode steps from other requests. Smooths out latency spikes
#     when long-context requests arrive in a hot batch.
#
#   --max-num-seqs 256
#     Concurrency ceiling per replica. Higher = more throughput but
#     more KV-cache pressure and higher per-token latency. Tune on
#     your real traffic shape rather than guessing.
#
#   --quantization fp8
#     Activate FP8 quantisation on H100-class hardware (supported by
#     vLLM via Marlin / FBGEMM-style kernels for FP8). Cuts memory
#     and improves throughput; verify quality on your evals first.

What is not shown: TLS termination (do it at the load balancer or sidecar, not in vLLM), authentication beyond the static API key (use a gateway), per-tenant rate limiting (also at the gateway), and request logging (mirror the request path through your observability stack).

Quantisation: How Far You Can Push the Memory Footprint

Quantisation is the single highest-leverage way to fit a larger model on smaller hardware. vLLM supports several quantisation formats — FP8 (on H100-class and newer hardware), INT8, AWQ, GPTQ, and GGUF-via-conversion. Each trades off in a different way between quality, throughput, and supported hardware.

FP8 is the modern default on H100, H200, B100/B200, and equivalent hardware. It is roughly transparent on quality for most workloads, halves the memory footprint versus BF16, and is well-supported in vLLM's kernel path. If you are on H100-class hardware, FP8 is usually the first thing to try.

AWQ and GPTQ are weight-only quantisation formats that compress to INT4 or INT3 weights with FP16 activations. They are common for running larger models on smaller GPUs — a 70B AWQ-int4 model fits on a single A100 80GB. The quality cost is workload-dependent; on commonsense knowledge tasks the regression is usually small, on tasks that stress long-tail factual recall it can be larger. The discipline is to benchmark the quantised model on your own evals — never assume the published numbers transfer.

INT8 occupies an awkward middle on modern hardware: less aggressive than INT4 weight-only formats, less throughput-optimised than FP8 on H100-class GPUs. It remains useful on A100 and earlier hardware where FP8 is unavailable.

The rule we apply: try the unquantised model first if it fits, fall back to FP8 on H100-class hardware before reaching for weight-only quantisation, and reserve INT4 (AWQ / GPTQ) for the cases where the smaller footprint is genuinely necessary. Quantisation introduces a quality variable that is easy to forget when debugging downstream issues months later.

Performance Levers Worth Knowing

Observability and Operability for vLLM in Production

Once vLLM is serving real traffic, the metrics you need are the same shape as any other inference platform but with model-specific specifics. vLLM exposes a Prometheus metrics endpoint that includes time-to-first-token (TTFT), inter-token latency, requests in queue, KV-cache utilisation, prompt and generation token counts, and preemption counts. Scrape it, dashboard it, alert on it.

The metrics most worth alerting on are: KV-cache utilisation sustained above ~95% (you are running close to the concurrency ceiling — either add replicas or drop max-num-seqs), preemption rate above zero (vLLM is having to evict in-flight requests to make room — same remediation), and TTFT p95 above the SLA you committed to (something has shifted upstream: longer prompts, more concurrency, hardware degradation).

For request-level visibility, the OpenAI-compatible endpoints emit standard token-count fields in the response, which slot neatly into LLM cost-tracking and evaluation pipelines. For request-level tracing across the agent stack, wrap vLLM client calls in OpenTelemetry spans the same way you would any other downstream service.

Model rollout discipline is its own topic. Self-hosted models do not have to track vendor release schedules, which is a feature — but it is also a feature you have to use deliberately. Stand up a canary deployment alongside the production deployment, shift a small fraction of traffic to it, compare metrics and eval scores against the production baseline for a few days, and only then promote. The model you are running is now part of your infrastructure; treat upgrades the way you would treat a database engine upgrade.

Where vLLM Tends To Be the Right Choice

Seven Decisions Worth Making Deliberately Before Your First vLLM Deployment

• **Model choice and quantisation tier.** Pick the smallest model that meets your quality bar — measured on your own evals, not on public leaderboards — and quantise only if the unquantised version does not fit. Quantisation is recoverable; over-committing to too large a model is expensive.
• **GPU class and topology.** H100 / H200-class hardware with NVLink is the path of least resistance for anything above 30B parameters. A100 / L40S-class works for smaller models and is cheaper. PCIe-only multi-GPU is a trap for tensor-parallel deployments — the interconnect is the bottleneck, not the compute.
• **Single replica vs horizontally scaled.** Production deployments need at least two replicas behind a load balancer for fault tolerance. Single-replica deployments are for staging and internal tools where outages are acceptable.
• **Memory utilisation ceiling.** `--gpu-memory-utilization` in the 0.85-0.92 band is the safe default. Higher values risk transient OOM under load; lower values waste capacity. Measure under peak load before committing.
• **Prefix caching and chunked prefill defaults.** Turn both on for any chat-style or RAG-style workload. The cases where you would turn them off are narrow and specific (workloads with no prefix overlap; latency-critical decode-only loops); default-on is the right starting point.
• **Observability scope from day one.** Wire the Prometheus metrics endpoint into your existing dashboards, alert on KV-cache utilisation and preemption rate, and instrument client-side traces through OpenTelemetry. Retrofitting observability after the first incident is much more expensive than building it in.
• **Model rollout process.** Decide how new model versions get promoted (canary deployment, traffic shift, eval gate) before you have to do your first upgrade. The model is now part of your infrastructure — give it the same change-management discipline you give every other dependency.

How To Approach a vLLM Pilot

A sensible first vLLM deployment is a non-critical workload with predictable traffic — internal knowledge-base search, a developer-facing assistant, a batch document-processing pipeline — running a small or medium open-source model that comfortably fits on a single GPU. Stand it up alongside whatever managed API the workload currently uses, mirror a fraction of traffic to it, compare quality and latency for a couple of weeks, and use that data to decide whether the economics justify expanding the footprint.

The assessment criteria worth measuring before promoting to the production path: quality on your own evals (not generic benchmarks), p50 / p95 / p99 latency under representative load, throughput per GPU dollar, operational incident rate during the pilot window, and the security and compliance review of the data path. If any of these is unsatisfactory, the right answer is usually to keep the workload on the managed API for now — self-hosting only pays back when you actually run the GPU hot.

If you are weighing self-hosted inference for a real workload — particularly one where managed-API costs are climbing, data residency is becoming a constraint, or you are running a fine-tuned model in production — our team can help scope it. The capability gap between managed APIs and vLLM has narrowed dramatically; the open question is usually unit economics and operational maturity, and that is a calculation specific to your traffic shape rather than a generic answer.