Buoyant open-sources a control plane for vLLM fleets
Kinference adds routing, rate limits, KV-cache awareness and tenant metrics above vLLM, targeting teams running many open models on Kubernetes.
The release
Buoyant, the company behind the Linkerd service mesh, has released Kinference, an Apache 2.0-licensed open-source control plane for inference workloads on Kubernetes. The project sits in front of vLLM workers and manages how requests are distributed across a fleet of model servers. Its initial release is v0.1.0-alpha.7 and is written in Rust.
Kinference brings several operational functions into one layer: policy-based routing, token-rate limits, KV-cache-aware load balancing and per-tenant metrics. It is configured through five Kubernetes custom resources, allowing platform teams to describe inference behavior in cluster-native terms rather than building a separate scheduler around every model endpoint.
Why the infrastructure layer matters
The release does not introduce a new model, but it addresses a growing bottleneck for open-weight deployment. Organizations increasingly run multiple models for different latency, privacy and cost requirements. Once those models share GPUs and serve different internal customers, the difficult problem becomes deciding which worker should receive each request, how much context a tenant can consume and how to keep cached attention states useful.
That is the territory Kinference is targeting. Its vLLM focus also makes the project relevant to teams that want to retain control over model serving instead of moving all inference to a proprietary cloud API. The Apache license lowers adoption friction, although the alpha status means production reliability, integrations and performance under failure still need independent validation.
Kinference is unlikely to change model rankings this week. Its significance is architectural: it treats inference as a fleet-management problem rather than a collection of isolated endpoints. If the project matures, a common control layer could make open models easier to operate at enterprise scale and intensify competition on deployment economics.
Uncle Cat take
Kinference’s strongest fact is its KV-cache-aware routing; that is more consequential for crowded vLLM fleets than another generic Kubernetes wrapper, but alpha software still carries the operational risk.