Kubernetes Is Evolving from an Application Orchestrator into an AI Workload Orchestrator

Kubernetes was designed to answer a relatively simple question: where should this container run?

AI is forcing it to answer a much harder one: where should this workload run, on which accelerator, alongside which other processes, with which model state, and where should the next request be routed?

For most of Kubernetes' history, applications looked reasonably predictable. You packaged an application into a container, declared how much CPU and memory it needed, created a Deployment, exposed it through a Service, and let Kubernetes handle the rest. That model works extremely well for stateless web applications.

AI workloads are different. A production AI platform may include GPU-backed inference servers, distributed training jobs, multiple model variants, LoRA adapters, KV caches, vector databases, retrieval pipelines, agent runtimes, sandboxes for generated code, long-running agent sessions, batch inference and external model providers.

None of those behave like ordinary HTTP services. And that difference is beginning to reshape Kubernetes itself.

The original resource model wasn't designed for this

The basic scheduling model is elegant. A Pod requests resources:

resources:
  requests:
    cpu: "2"
    memory: "4Gi"

The scheduler finds a node capable of running it. Two numbers, one decision.

Now look at the right-hand side of the diagram above. An AI workload wants a particular accelerator, a particular amount of high-bandwidth memory, a particular interconnect, placement within a rack, a specific model, a specific adapter, and - if you can manage it - a cache that is already warm.

Find a node with enough resources stops being sufficient. The scheduler needs to understand relationships between workloads, accelerators, topology and shared state.

That is why Dynamic Resource Allocation matters.

1. GPUs are becoming first-class resources

Historically, Kubernetes exposed GPUs through device plugins and extended resources:

resources:
  limits:
    nvidia.com/gpu: 1

That works. But the request means little more than give me one GPU. AI infrastructure increasingly needs something closer to give me an accelerator matching these characteristics, subject to these constraints and preferences.

Dynamic Resource Allocation provides that richer model. The core DRA APIs reached GA in Kubernetes 1.34, where it is also enabled by default. It is stable, and the project has committed to no breaking changes.

The shape of it will look familiar:

This is the same trick storage pulled years ago. You stopped saying mount disk 3 and started saying give me 100Gi of fast SSD, and something else worked out which disk. nvidia.com/gpu: 1 is the old sentence. DRA is the new one.

It matters most when clusters are heterogeneous. A workload can express a preference order - an H100 if one is free, an A100 otherwise, some other compatible accelerator failing that - rather than the cluster operator hard-coding placement into node selectors.

Kubernetes is beginning to understand hardware as something workloads describe semantically, rather than an integer attached to a node.

2. Scheduling a Pod is not the same as scheduling a workload

Traditional Kubernetes scheduling is Pod-oriented. Distributed training exposes the limitation immediately.

Scheduling three workers is not 37.5% of a training job. It is zero training and three idle GPUs you are paying for. The workload needs an all-or-nothing guarantee: schedule the group together, or don't schedule it yet. That is gang scheduling.

Placement matters just as much. Eight GPUs in one rack on a fast interconnect will outperform eight spread across a datacentre, sometimes dramatically. Network topology becomes part of the scheduling decision rather than an implementation detail beneath it.

Kubernetes v1.36 advanced this considerably. It separates the Workload API as a static template from a new PodGroup API handling runtime state, and adds a PodGroup scheduling cycle to kube-scheduler enabling atomic workload processing. It also debuts the first iterations of topology-aware scheduling, which co-locates a PodGroup into a single topology domain, and workload-aware preemption, which treats a PodGroup as a single preemptor unit rather than evaluating its Pods in isolation.

The scheduler is evolving from where can this Pod run? toward where can this entire workload run efficiently?

3. Round-robin load balancing starts breaking down

Scheduling is only half the problem. Once model servers are running, requests have to reach them.

For a traditional REST API, balancing across healthy replicas is usually good enough, because the replicas are genuinely interchangeable. Model servers are not.

Round robin picks GPU A. Least-connections picks GPU A too, because A has the shortest queue. Both are wrong, because neither knows that GPU C already has adapter Z resident in memory. Sending the request to A means loading the adapter from scratch - a cold start that can dwarf the queue time you saved.

The replicas are not interchangeable. They differ by what is currently loaded in GPU memory, and the load balancer cannot see it.

Routing therefore has to become inference-aware, taking account of queue depth, GPU utilisation, KV-cache state, model availability, loaded adapters, request priority and expected generation load.

This is exactly what the Gateway API Inference Extension targets. The project's own announcement described traditional HTTP and round-robin approaches as insufficient for many generative-AI inference workloads.

4. Meet InferencePool

The key new abstraction is InferencePool - a collection of model-serving Pods sharing characteristics such as accelerator configuration, model server and base model. It reached stable in Gateway API Inference Extension v1.0, driven by WG-Serving and SIG Network.

That is a subtle but fundamental change. The network layer is no longer simply routing HTTP requests. It is participating in AI workload optimisation - and it is doing so with information that previously lived only inside the model server.

5. The gateway is becoming an AI gateway

The trend goes beyond endpoint selection. In March 2026 the Kubernetes community announced an AI Gateway Working Group, focused specifically on networking infrastructure for AI workloads.

The working group's scope includes token-based rate limiting, payload inspection, intelligent caching, semantic routing, guardrails and secure egress to external AI providers.

Notice what changes. An ordinary gateway counts requests and routes on the URL. An AI gateway counts tokens and routes on what the request means. Rate limiting by requests per second is close to meaningless when one request might generate four tokens and the next forty thousand.

Ingress → Service → Pod does not have vocabulary for any of that.

6. Agents introduce yet another workload shape

Then agents arrive, and they are close to the inverse of everything a Deployment assumes.

An agent may need a persistent workspace, filesystem state, a stable identity, tool credentials, isolation, suspend and resume, and somewhere safe to execute code a model just wrote. Those needs come from what an agent runtime holds inside it; the reason there are suddenly many such agents to schedule comes from agents learning to delegate to each other.

You can approximate that today by assembling a StatefulSet, a PVC, a Service, a NetworkPolicy, a RuntimeClass and a pile of custom lifecycle logic. It works. It is also a lot of machinery to reproduce for every agent.

Kubernetes SIG Apps is developing Agent Sandbox, which launched as a subproject at KubeCon NA 2025 and introduces three primitives: Sandbox as the core workload resource, SandboxTemplate as the security blueprint, and SandboxClaim for requesting an execution environment. It also explores warm pools, so a suspended agent resumes quickly instead of cold-starting.

This is the clearest signal in the whole post. AI is not simply running inside Kubernetes. AI is changing what a Kubernetes primitive looks like.

7. The observability model changes too

CPU, memory, requests per second, latency and error rate don't disappear. They just stop being sufficient.

Inference brings time to first token, time per output token, tokens per second, GPU and HBM utilisation, queue depth, KV-cache hit rate, batch size and model loading time. Agents bring execution time, tool calls, sandbox utilisation, token consumption, task completion and retries. Distributed training brings worker synchronisation, network bandwidth, checkpoint status, throughput and accelerator failures.

A platform team needs visibility across all seven layers, not just application → pod. This is why the AI platform stack around Kubernetes is becoming an infrastructure discipline of its own.

8. Cost becomes a scheduling problem

There is a blunter reason all of this matters. GPUs are expensive.

Every pod is Ready. Every health check passes. Every dashboard is green. And roughly two-thirds of the accelerator spend is doing nothing.

In traditional infrastructure, autoscaling mostly answers do I have enough capacity? In AI infrastructure it increasingly answers am I using the capacity I already pay for?

That pulls scheduling, batching, inference routing, autoscaling, KV-cache locality, model placement and hardware selection into a single optimisation problem. They are not separate concerns any more, because they all move the same number.

The new Kubernetes AI stack

Put it together and a modern Kubernetes AI platform looks less like application on Kubernetes and more like a layered system where Kubernetes understands a great deal about what sits above it.

What platform engineers should learn next

If you're already comfortable with Kubernetes, I wouldn't start by memorising another collection of AI frameworks. I'd learn where AI changes infrastructure assumptions.

The order matters. The top of that table is production-ready today; the bottom tells you where things are heading. Both are worth knowing, for different reasons.

Kubernetes isn't being replaced by the AI stack

Every major computing shift eventually produces the same question: do we need a completely new infrastructure platform?

So far, AI is producing a different answer.

Instead of Kubernetes disappearing, Kubernetes is absorbing the new workload requirements - accelerator orchestration, workload-aware scheduling, inference-aware networking, model routing, agent isolation, state management and AI-specific observability. It is acquiring an abstraction for each.

The interesting part isn't that companies are running AI on Kubernetes. We've been doing that for years. The interesting part is that AI is changing Kubernetes itself.

Kubernetes started life answering where should my container run? The AI era requires it to answer where the workload should run, which accelerator it should use, where its model requests should go, what state should stay warm, and how the whole system can run efficiently.

That is a much bigger job. And it suggests the next phase of Kubernetes may not primarily be about container orchestration at all.

Kubernetes is evolving into the control plane for AI infrastructure.

This post is about the layer underneath. Its companions work upward from there: the model is not your agent is about the runtime inside a single agent, and MCP gave agents tools, A2A gives agents colleagues is about how independent agents delegate work to one another.