In Part 1 of AI Infrastructure for Cloud Engineers, we looked at why Kubernetes is becoming an important foundation for production AI systems.
Read Part 1: Why Kubernetes Is Becoming the Operating System for AI Infrastructure
Now the question is: what actually changes when we start running AI workloads on Kubernetes?
Traditional applications are usually scheduled around familiar resources such as CPU, memory, storage, and network capacity.
AI workloads introduce another resource that changes the infrastructure equation:
GPU
GPUs are powerful, expensive, and limited. Once teams start running model inference, embedding services, fine-tuning jobs, or other AI workloads at scale, simply deploying a container is no longer enough.
The platform also needs to decide:
- Which workload gets which GPU?
- How should those workloads be scheduled?
- When should inference capacity scale?
- How do we avoid leaving expensive accelerators idle?
- How should models be served reliably in production?
This is where Kubernetes scheduling, autoscaling, and model-serving patterns start to matter.
Let's look at how these pieces fit together.
1. How Kubernetes Sees a GPU
Kubernetes normally schedules Pods based on resources such as CPU and memory.
A basic application might request:
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "1"
memory: "2Gi"
GPU-enabled nodes add specialized resources to the cluster.
A workload can then request a GPU:
resources:
limits:
nvidia.com/gpu: 1
Conceptually, the cluster might look like this:
Kubernetes Cluster
Node A
├── CPU
├── Memory
└── No GPU
Node B
├── CPU
├── Memory
└── GPU
Node C
├── CPU
├── Memory
└── GPU
If an AI workload requests a GPU, Kubernetes needs to place it on a node where that resource is available.
Hardware vendors commonly expose devices such as GPUs to Kubernetes through mechanisms including device plugins.
That sounds straightforward.
At larger scale, however, GPU scheduling becomes much more interesting.
2. Not Every GPU Is the Same
Imagine a cluster with multiple accelerator types.
Node A → NVIDIA T4
Node B → NVIDIA A100
Node C → NVIDIA H100
Node D → CPU only
Now imagine three workloads:
Small embedding model
Large language model
Distributed training job
Placing all three randomly would be inefficient.
The embedding workload may not need the most powerful GPU, while the large model may require significantly more accelerator memory and compute.
This means AI platforms often need to consider:
GPU type
GPU memory
Workload size
Topology
Availability
Cost
Priority
Modern Kubernetes scheduling is evolving specifically for these kinds of workloads.
Kubernetes v1.36, for example, introduced further workload-aware scheduling capabilities including PodGroups, topology-aware scheduling, workload-aware preemption, and integration with Dynamic Resource Allocation. These features are particularly relevant to tightly coupled AI/ML and batch workloads.
3. Why Workload-Aware Scheduling Matters
Traditional Kubernetes scheduling largely thinks about individual Pods.
AI workloads may need Kubernetes to think about a group of Pods together.
Imagine distributed training that requires four workers:
Training Job
Worker 1
Worker 2
Worker 3
Worker 4
Scheduling only two workers while the others remain pending may not be useful if the job requires all four before it can start.
This is the idea behind gang scheduling.
Enough resources for all workers?
Yes
↓
Schedule workload
No
↓
Wait for capacity
Topology can also matter.
If several workers constantly exchange large amounts of data, placing them far apart across the infrastructure may introduce unnecessary network overhead.
Workload-aware and topology-aware scheduling allow Kubernetes to make placement decisions using more context about the complete workload rather than treating every Pod independently.
4. GPU Utilization Matters
GPUs can represent a significant portion of the infrastructure cost behind self-hosted AI.
That makes low utilization expensive.
Imagine:
GPU Capacity
████████████████████ 100%
Actual Workload
██████ 30%
The remaining capacity is still being paid for.
This can happen when:
- Models are oversized for the hardware
- Traffic is inconsistent
- Each workload reserves an entire accelerator
- Requests are not batched efficiently
- Too many replicas are running
- GPUs are fragmented across teams
One goal of an AI platform is therefore not just:
Make the model run.
It is:
Keep the model responsive while using expensive compute efficiently.
The cloud-native ecosystem is increasingly developing GPU-sharing and accelerator-aware scheduling approaches for this reason. For example, HAMi focuses on sharing and scheduling heterogeneous accelerator resources, while Kubernetes Dynamic Resource Allocation provides a more flexible mechanism for requesting specialized devices.
5. Inference Is Different From Training
AI infrastructure discussions often combine training and inference, but they have different operational characteristics.
Training
Training commonly looks like:
Dataset
↓
Training Job
↓
Many GPUs
↓
Hours / Days
↓
Model
The workload may require several accelerators simultaneously and run for a long period.
Inference
Inference looks more like:
User Request
↓
Model Server
↓
GPU
↓
Generated Response
Inference is usually much more sensitive to:
Latency
Throughput
Availability
Queue depth
Concurrent requests
For a user-facing AI application, a model that eventually returns the correct answer is not enough.
It also needs to respond within an acceptable amount of time.
That changes how we think about scaling.
6. CPU Is Not Always the Right Scaling Metric
For many web applications, Kubernetes autoscaling might use CPU utilization.
CPU > 70%
↓
Add Pods
That can work well for traditional services.
AI inference may need different signals.
Imagine an inference server where:
CPU = 35%
GPU = 92%
Waiting requests = 120
From CPU alone, the application may appear healthy.
From the user's perspective, it may already be overloaded.
Better AI scaling signals may include:
GPU utilization
Requests waiting
Concurrent requests
Inference latency
Tokens per second
KV cache utilization
Queue depth
KServe, for example, supports autoscaling inference workloads using external LLM metrics through technologies such as KEDA, Prometheus, and OpenTelemetry. Its documentation includes examples based on active or waiting inference requests rather than relying only on CPU.
A simplified scaling flow could look like this:
Request Queue
↓
Waiting requests increase
↓
Autoscaling signal
↓
Create more inference replicas
↓
More capacity available
7. What Model Serving Actually Means
A trained model is essentially an artifact.
Users still need a service capable of loading the model and accepting requests.
That layer is commonly called model serving.
Conceptually:
Application
↓
Model Endpoint
↓
Inference Server
↓
Model
↓
GPU
A production model-serving layer may need to handle:
- Model loading
- Request routing
- Batching
- Autoscaling
- Health checks
- Model versions
- Metrics
- GPU allocation
- Failure recovery
Instead of application developers building all of this independently, model-serving frameworks can provide reusable infrastructure.
One Kubernetes-native example is KServe, which provides abstractions for deploying and operating inference workloads on Kubernetes.
The wider cloud-native ecosystem is also building more specialized inference infrastructure. Kubernetes' former WG Serving helped advance inference-oriented capabilities including request scheduling and gateway patterns before concluding its work in 2026.
8. A Simplified Kubernetes AI Architecture
Putting the pieces together, an inference platform might look like this:
Users
↓
API / AI Gateway
↓
Request Router
↓
┌───────────┼───────────┐
↓ ↓ ↓
Model Pod Model Pod Model Pod
↓ ↓ ↓
GPU GPU GPU
Kubernetes Cluster
↓
┌─────────────┼─────────────┐
↓ ↓ ↓
Autoscaling Monitoring Scheduling
Kubernetes handles the infrastructure layer.
The model-serving layer handles inference-specific concerns.
Together, they allow the platform to respond to changing demand.
9. Model Routing Is Becoming Smarter
Basic load balancing assumes that multiple application replicas are roughly interchangeable.
AI inference can be different.
The best place to route a request may depend on:
- Which model is already loaded
- Available GPU memory
- Current request queue
- Cached model state
- Accelerator type
- Current inference load
So instead of:
Request
↓
Random Pod
AI-aware routing can move toward:
Request
↓
Inference Gateway
↓
Best available model server
Modern cloud-native inference projects are increasingly exploring model-aware and state-aware routing.
For example, llm-d focuses on capabilities such as inference scheduling, KV-cache-aware behavior, and separating prompt processing from token generation to improve resource utilization and inference performance.
This is one of the clearest examples of Kubernetes infrastructure adapting specifically to AI workloads.
10. Scaling the Pod Does Not Scale Everything Else
There is an important limitation to remember.
Suppose Kubernetes scales an inference application:
2 replicas
↓
4 replicas
↓
8 replicas
Those eight replicas may now generate much more traffic toward:
Vector database
Object storage
External APIs
Model storage
Network
GPU nodes
Scaling one component can simply move the bottleneck somewhere else.
For example:
Inference Pods
████████████████ Healthy
↓
Vector Database
████████████████ Overloaded
Capacity planning needs to consider the complete request path.
This is the same lesson cloud engineers already know from distributed systems.
AI does not remove bottlenecks.
It introduces some new ones.
11. Failure Handling Still Matters
Suppose a GPU node fails while running an inference workload.
The platform needs to detect that condition and recover.
A production architecture should consider:
Pod failures
Node failures
GPU failures
Model loading failures
Provider failures
Network failures
Out-of-memory conditions
Kubernetes can restart or reschedule workloads, but AI platforms also need visibility into accelerator health and inference behavior.
Recent Kubernetes Dynamic Resource Allocation work includes exposing device health information to workloads and controllers, which can help operators understand failures involving specialized hardware.
The key point is simple:
Process running
≠
AI service healthy
Infrastructure health and application health both matter.
12. What Should We Monitor?
For an AI workload running on Kubernetes, I would separate metrics into three layers.
Kubernetes
Pod availability
Pod restarts
Node health
CPU
Memory
Network
GPU
GPU utilization
GPU memory
Accelerator availability
Device health
Inference
Request latency
Queue depth
Requests running
Tokens per second
Time to first token
Inference errors
Looking at only one layer can hide the real problem.
For example:
Kubernetes
Pods healthy ✓
GPU
Utilization 100%
Inference
Latency increasing ↑
Queue growing ↑
The cluster is technically running.
The service is still degrading.
What Cloud Engineers Should Take Away
You do not need to become a machine-learning researcher to work with AI infrastructure.
The infrastructure problems remain very familiar:
Scheduling
Scaling
Networking
Capacity
Observability
Security
Reliability
Cost
The difference is the resource being managed.
Instead of only asking:
How much CPU?
How much memory?
we now also ask:
Which GPU?
How much GPU memory?
Which model?
How many concurrent requests?
How many tokens per second?
Where should this inference request run?
That is the bridge between traditional cloud engineering and AI infrastructure.
Practical Architecture Checklist
Before running AI workloads on Kubernetes, think about:
- Which workloads actually require GPUs
- Which accelerator types each workload needs
- Whether workloads need single or multiple GPUs
- How GPU nodes are isolated from general workloads
- How models are served
- Which metrics trigger scaling
- How requests are routed
- How GPU utilization is measured
- Whether several workloads can share accelerator capacity
- What happens when a GPU or node fails
- Whether downstream dependencies can handle additional replicas
- How much idle accelerator capacity you are paying for
What's Next?
GPUs, scheduling, and model serving solve only part of the production problem.
Once the application is running, the next question becomes:
How do we know whether the AI system is actually healthy?
Traditional infrastructure monitoring gives us CPU, memory, and Pod health.
AI workloads introduce another set of signals including inference latency, token throughput, GPU utilization, queue depth, model failures, and cost.
That is what we will cover next.
Part 3: Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory
Final Thoughts
Running AI workloads on Kubernetes is not simply a matter of adding a GPU to a Pod.
Production systems need to think about the complete lifecycle:
GPU Allocation
↓
Scheduling
↓
Model Serving
↓
Request Routing
↓
Autoscaling
↓
Observability
↓
Failure Recovery
Kubernetes gives us a strong orchestration foundation.
But AI introduces new constraints around expensive accelerators, workload placement, inference latency, and resource utilization.
The interesting shift is that Kubernetes is beginning to understand more about these workloads directly, while projects around it are adding the inference-specific capabilities required to operate AI efficiently.
For cloud engineers, this is where existing Kubernetes knowledge starts becoming directly useful in the AI infrastructure world.
Thanks for Reading
This article is Part 2 of my AI Infrastructure for Cloud Engineers series:
- Why Kubernetes Is Becoming the Operating System for AI Infrastructure
- Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving
- Coming next: Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory
- FinOps for AI: Understanding GPU, Token, and Inference Costs
- Building a Production AI Platform: Kubernetes, GitOps, IaC, Security, and Observability
I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, and the engineering behind production AI systems.
Looking forward to connect, learn and grow together 😄
LinkedIn: Connect with me on LinkedIn
If you're running AI workloads on Kubernetes, what has been harder in practice: GPU allocation, autoscaling, model serving, or keeping the GPUs efficiently utilized?
