llm-d 1.0 coordinates distributed inference for agent workloads

Agent traffic exposes the limits of round-robin serving

Google and more than 500 contributors across 25 organisations have released llm-d 1.0, an open-source Kubernetes-native stack for distributed language-model inference. The CNCF Sandbox project connects serving engines such as vLLM and SGLang with cluster orchestration. It is designed for multi-step agents and reinforcement-learning rollouts that reuse long prefixes, fan out across tools and arrive in bursts.

A generic load balancer can send each turn to a different replica, forcing the same context to be recomputed while accelerator memory fills with duplicate key-value cache. One busy agent loop can also delay interactive users. llm-d coordinates model servers as a serving pool so routing, cache state and traffic priority can be managed across the fleet.

Version 1.0 establishes a production contract

The release introduces stable version-one APIs and a hardened request path tested across clouds and accelerator types. It provides priority and fairness-aware routing for multi-tenant quality of service, router high availability and zero-downtime updates. DisaggregatedSet coordinates rolling updates across separate prefill and decode pools so active sessions do not lose context.

Stability does not remove upgrade risk. Operators should pin charts, engines and model versions, replay multi-turn traffic before promotion and test failure during each rollout phase. A router that knows where context resides becomes critical state; high availability and recovery should be exercised under real cache and session loads rather than assumed from control-plane health.

Routing remembers where context already lives

The llm-d router tracks cached prefixes while weighing replica load and predicted latency. When accelerator memory fills, connectors can move cache to host memory, SSD or Google Cloud Managed Lustre rather than immediately discarding it. Google reports that prefix-aware routing with host-memory offload cut median time to first token by more than 85 per cent and lifted throughput 2.4 times on one Ironwood TPU coding workload.

Those figures are workload-specific, but the mechanism addresses a real source of waste. Teams should measure cache hit rate, offload bandwidth, restoration latency and storage cost together. A cache tier that improves average throughput may still create long-tail stalls when many sessions recall large contexts at once.

Prefill, decode and background work get separate controls

Prefill is compute-heavy while token decoding is often memory-bound, so llm-d can place them in separate pools and scale each independently. It also introduces an asynchronous processor for queued work from systems such as Pub/Sub or Redis. Priority lanes and quotas allow batch requests to use spare capacity without starving real-time traffic.

Google says time-slicing can interleave asynchronous inference into idle periods during reinforcement learning, raising aggregate accelerator duty cycles from around 40 per cent to above 70 per cent in its test. Capacity gains need guardrails: a background queue should back off predictably, expose waiting time and never borrow resources required for an interactive service-level objective.

One endpoint can span TPUs and GPUs

llm-d can place TPU and GPU pools behind a single endpoint, with TorchTPU allowing PyTorch models to run natively in vLLM and SGLang on Google accelerators. Agents need not change when capacity moves between pools. Autoscaling can use queue depth, saturation and predicted latency, while quotas control shared accelerator access.

Mixed hardware increases choice but complicates reproducibility. Numerical behaviour, quantisation support and kernel performance can differ between pools. Platforms should record where a response ran, validate equivalent quality and prevent the router from moving regulated or licensed models to an unauthorised accelerator environment.

Open benchmarks help separate claims from configurations

Prism provides dashboards and standard recipes for comparing throughput, latency and cost. Google also cites production results including higher prefix-cache hit rates and lower time to first token, plus customer reports from Snap and ZenteiQ. These results are useful because distributed inference performance depends heavily on prompt length, concurrency, model shape and cache reuse.

Teams can begin with prefix-aware routing in front of an existing deployment, then add flow control, offload and disaggregation only where measurements justify the complexity. llm-d 1.0 is less a new model than an operating layer for expensive model fleets. Its success will be measured by predictable service under contention, safe upgrades and transparent economics rather than peak benchmark throughput alone.

Operational test

Before production use, teams should document the exact product or model version, supported region, source date and evaluation conditions. They should repeat representative tests after any model, compiler, connector, policy or infrastructure change. Published capability and performance claims are useful starting points, but confidence must come from evidence collected in the organisation’s own environment. A named owner, measurable acceptance threshold, incident path and tested fallback turn a promising announcement into a service that can be governed over time.

Evaluation should cover ordinary work, boundary cases and deliberate misuse rather than a polished demonstration alone. Results need to be sliced by language, user role, data sensitivity and consequence so a strong average cannot hide a serious failure. Teams should also estimate the human work required to review exceptions and maintain the surrounding controls. If that effort is omitted, an apparently efficient feature may simply move cost and risk into operations, security or frontline staff who were not included in the original business case.