llm-d 1.0 Explained: Kubernetes-Native Distributed LLM Inference

llm-d sits around inference engines and helps a cluster make coordinated decisions about where requests should run, how cached context can be reused, how competing traffic is prioritized, and how serving components are operated. Its design connects model-serving engines such as vLLM and SGLang with Kubernetes-native scheduling and deployment practices. The project is open source, and its components can be composed instead of adopted as one indivisible product.

That distinction matters. vLLM or SGLang still performs model execution; llm-d adds distributed routing and operational behavior around those engines. Kubernetes provides the surrounding platform primitives, while a cloud provider may offer a managed integration. Google Cloud, for example, describes GKE Inference Gateway as a managed way to run the llm-d router in its GKE environment. That managed service is not the same thing as the upstream open-source project.

The project remains a CNCF Sandbox project. Version 1.0 is a meaningful release milestone, but it does not mean every component is a Kubernetes standard, every accelerator combination has identical support, or production outcomes are guaranteed without workload-specific testing. Teams should verify the engine, accelerator, storage connector, routing policy, and deployment recipe they intend to operate.

If the application uses retrieval-augmented generation, our guide to building production-ready RAG systems covers the application-side retrieval and evidence pipeline that sits above model serving.

A short chat request often looks like one prompt followed by one answer. An agent may instead read files, call tools, inspect results, retry a failed step, and ask the model to reason again. Several agents can do this at once, and each step may reuse a long prefix from the same conversation or task. The resulting workload has repeated context, uneven request sizes, unpredictable bursts, and a mixture of latency-sensitive interactions and background work.

The key-value (KV) cache stores intermediate attention state that lets a model server avoid recomputing portions of a prompt already processed. That state consumes accelerator memory, and it is useful only when the request reaches a server that can reuse it or when the cache can be transferred efficiently. A generic round-robin load balancer may distribute requests evenly while sending them to replicas that must rebuild the same context.

Cache locality alone is not enough. If every request is sent to the replica with the warmest cache, that replica can become overloaded while less-busy replicas sit idle. A production router has to balance reuse against current load and latency. Routing policy should be evaluated with the actual prompt lengths, concurrency, and arrival patterns of the application rather than a toy request loop.

Application-side orchestration and failure handling are explored in our production AI-agent guide; llm-d addresses the serving infrastructure beneath those agent steps, not the agent's task logic.

At a high level, a request enters through a serving endpoint and reaches a routing layer. The router can consider whether a candidate replica holds a matching prefix, how busy it is, and which traffic priority applies. It then forwards the request to a model-serving engine. Depending on the deployment, the stack can also use separate pools for prefill and decode, plus additional cache tiers when accelerator memory is not sufficient for the working set.

Prefill processes input tokens and builds the attention state needed for generation. Decode then produces output tokens incrementally and is often constrained by memory bandwidth and active sequence state. The two phases have different resource profiles, so keeping them together is not always the best use of a large fleet. Disaggregated serving lets operators scale and place the phases separately, but introduces networking, coordination, and version-compatibility requirements that must be measured.

KV-cache tiering extends the memory hierarchy rather than making memory unlimited. Frequently reused state may remain in accelerator memory; other state can be moved to host memory or supported storage tiers. Moving it has a cost in transfer latency and bandwidth, so offloading is most useful when the alternative is expensive recomputation or when the working set exceeds the fastest tier.

The October 20 release notes emphasize a stable v1 configuration API and a hardened request path, along with high-availability routing and safer upgrades. For disaggregated serving, the project describes coordinated management of prefill and decode replicas so a rolling update does not casually mix incompatible revisions or drop active context. Those operational concerns become critical once a serving endpoint supports real users.

The release also puts more emphasis on multi-tenant quality of service. Priority- and fairness-aware flow control can help keep background agent loops from consuming all available capacity when interactive traffic arrives. Session-aware routing and turn-priority scheduling are intended to preserve useful context across multi-step interactions. The practical effect depends on how a team configures priorities, admission limits, timeouts, and overload behavior.

The ecosystem includes cloud-specific paths as well as upstream components. Google Cloud describes GKE integration, mixed Google TPU and GPU pools, and managed routing. Those details are relevant to teams already using that environment, but should not be read as a guarantee that every managed feature is available unchanged on every Kubernetes distribution. Use the project's supported deployment recipes and the relevant provider documentation for the target.

The Google Cloud release announcement reports workload-specific gains from its own deployments and benchmarks. For example, it describes a coding-agent workload on Ironwood TPU where prefix-aware routing plus host-memory offload reduced median time to first token by more than 85% and increased throughput 2.4 times versus a plain Kubernetes Service. It also describes a multi-region deployment benchmark. These figures are useful clues about workloads that may benefit, but they are not an independent, cross-platform benchmark of every llm-d configuration.

Results depend on the baseline, model, prompt distribution, output lengths, concurrency, accelerator generation, cache state, and definition of success. A deployment that is decode-bound, has little reusable context, or cannot keep the cache warm may see a different outcome. The extra router and storage layers can also add overhead when a workload is small or already well served by a single engine.

Before repeating a headline number in a business case, read the linked benchmark configuration and reproduce the closest workload on the target hardware. Report latency and throughput together, and include cost, failures, and tail behavior. If a result improves the median but worsens p95 latency or increases errors under burst, it may not satisfy the service-level objective that matters to users.

For a repeatable evaluation plan, see our LLM evaluation pipeline guide, which explains frozen cases, slice analysis, regression checks, and release gates.

llm-d is most compelling when a team operates its own model-serving fleet, has enough concurrency or context reuse for distributed routing to matter, and needs controls that a basic load balancer does not provide. It is also more attractive when the organization already understands Kubernetes operations and can support the extra metrics, policies, and rollout procedures. Teams using a hosted model API may have no reason to introduce this layer because the provider already owns the serving infrastructure.

A small deployment with one model, modest traffic, short prompts, and no strict multi-tenant requirements may be simpler to operate with a single engine and a conventional service. The right baseline is not an outdated architecture chosen to make a new system look fast; it is the simplest supported system that meets the workload's real service objectives.

Write down the reason for evaluating llm-d before running a benchmark. Examples include repeated long prefixes causing excess prefill work, accelerator memory pressure, poor tail latency under bursts, the need to separate prefill from decode, or a requirement to protect premium traffic from background jobs. If the current system does not exhibit one of these problems, the extra control plane may add complexity without enough benefit.

Choose one workload and one supported deployment recipe. Capture the current baseline before changing the router, cache tier, or model-server topology. Freeze representative prompts and conversation traces, including long-context requests, short interactive turns, bursty arrivals, and failure cases that currently cause incidents. Record the model revision, engine version, hardware, concurrency, routing configuration, and cache state so another engineer can reproduce the run.

Track time to first token (TTFT), inter-token latency or time per output token, end-to-end latency, completed tokens per second, queueing delay, error and timeout rates, cache-hit behavior, accelerator utilization, and cost per useful request. Break results down by prompt length, output length, tenant, and priority class. Keep cold-cache and warm-cache tests separate; a mixed average can hide a configuration that only works after a favorable warm-up.

Exercise operational behavior before routing important traffic through the new path. Test a router restart, a model-server replacement, cache eviction, storage slowdown, queue saturation, and a rolling update. Confirm authentication and tenant isolation are enforced by trusted services, not inferred from model-generated arguments. Set explicit limits and rollback thresholds, and make sure dashboards can identify which routing or serving decision caused a regression.

A cache-aware router can overload a warm replica if affinity is treated as an absolute rule. Operators need a load or saturation signal that lets the router choose a less-local endpoint when the warm one is too busy. Offloading can trade recomputation for network and storage latency, so the best cache tier depends on the actual reuse window and the speed of the underlying media.

Disaggregation can improve resource matching, but it is not free parallelism. It adds inter-pool communication and more components to deploy, observe, and upgrade together. A poor rollout policy can put incompatible prefill and decode revisions into the same request path. High availability also needs failure drills: a configured standby is not proof that failover preserves required latency and session behavior.

The stack does not solve model quality, prompt design, retrieval relevance, application authorization, or unsafe tool behavior by itself. Those concerns remain in the application and policy layers. The inference platform should be evaluated as one component of a larger production system, with independent tests for correctness, security, and user outcomes.

A sensible first step is to deploy a well-lit baseline using a supported engine and hardware path, then measure the request profile without additional optimizations. Add prefix-aware routing when cache reuse is the bottleneck. Introduce flow control when tenants compete for capacity, and consider offloading or disaggregation only after the baseline demonstrates a memory or phase-specific constraint. Each step should have its own success criteria and rollback plan.

Teams building applications around models should keep the serving layer separate from business workflow. llm-d can help a cluster deliver model tokens, but application code still owns request validation, data access, tool permissions, retries, and user-facing failure handling. Our guide to connecting LLMs to APIs and business workflows covers those application integration responsibilities.

The central lesson of llm-d 1.0 is operational: model serving becomes a distributed-systems problem when context, traffic, and tenants grow. Cache locality, load, fairness, upgrades, and observability have to work together. Adopt the parts that address a measured constraint, reproduce results on your own workload, and retain the simplest architecture that meets your service objectives.