Buoyant launches Kinference, an open-source control plane for vLLM inference on Kubernetes
The Apache 2.0 project adds KV-cache-aware load balancing, per-tenant rate limiting and usage tracking, with vendor-claimed TTFT reductions of up to 95 percent.
Buoyant, the company behind Linkerd, has introduced Kinference, an open-source control plane for building production inference platforms on top of vLLM in Kubernetes. The project, published under the Apache 2.0 license on 4 November, provides token rate limiting, per-tenant usage and cost tracking, intelligent load balancing, and inference policies defined through Kubernetes custom resources.
Its differentiating feature is a load balancer that reads the contents of GPU KV caches through NVIDIA's dynamo-kv-router, steering each request toward the worker that already holds the relevant state. On one benchmark workload, Buoyant reports median time-to-first-token falling by 95 percent and median end-to-end latency by 45 percent compared with standard Kubernetes load balancing, figures that remain vendor claims pending independent reproduction. The project ships with an OpenAI-compatible endpoint and active GitHub documentation, positioning open-source infrastructure as an alternative to commercial inference platforms for multi-tenant deployments.