Thanks for taking the time to read. If you’ve worked with AI infrastructure or observability, I’d love to hear your experience in the comments.
In Part 2 of AI Infrastructure for Cloud Engineers, we looked at how GPUs, scheduling, autoscaling, and model serving change the way AI workloads run on Kubernetes.
Read Part 2: Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving
Getting an AI workload into production is only the beginning.
Once users depend on it, the questions change:
- Is the platform healthy?
- Are GPUs being used efficiently?
- Why is inference slowing down?
- Are requests building up in a queue?
- Is the model producing tokens fast enough?
- Which dependency is causing the delay?
A Kubernetes dashboard showing healthy Pods cannot answer all of these questions.
Production AI systems need visibility across infrastructure, accelerators, inference, and the full request path.
Why CPU and Memory Aren’t Enough
Traditional infrastructure metrics still matter.
For Kubernetes, we still need to watch:
CPU
Memory
Pod availability
Pod restarts
Node health
Network
Storage
Request rate
Error rate
Latency
But consider this:
Pods Running: 5/5
CPU Usage: 42%
Memory Usage: 58%
Pod Restarts: 0
Everything looks healthy.
Now look at the AI workload:
GPU utilization: 99%
Inference latency: Increasing
Queue depth: Growing
Time to first token: Increasing
The platform is technically running.
The user experience is still getting worse.
That is the main difference with AI infrastructure observability: you need to connect infrastructure health with model behavior.
The 4 Layers of AI Observability
I find it useful to think about AI observability in four layers:
1. Kubernetes Infrastructure
↓
2. GPU / Accelerator
↓
3. Model Inference
↓
4. End-to-End Request
Each layer answers a different question.
1. Kubernetes Infrastructure
This is the foundation.
Monitor:
Pod availability
Pod restarts
Node health
CPU utilization
Memory utilization
Network
Storage
Deployment health
If inference suddenly becomes slow, you first need to know whether the problem is actually inside the model.
Maybe the Pod is under memory pressure.
Maybe a node has a networking issue.
Maybe the application cannot reach a dependency.
Kubernetes metrics provide that first layer of context.
2. GPU and Accelerator Metrics
GPUs are often among the most expensive resources in an AI platform.
Useful signals include:
GPU utilization
GPU memory usage
Temperature
Power consumption
Device health
GPU errors
For NVIDIA environments, DCGM Exporter can expose GPU telemetry in a Prometheus-compatible format.
A simple monitoring flow might look like this:
GPU Nodes
↓
DCGM Exporter
↓
Prometheus
↓
Grafana
The goal is not just to ask:
Is the GPU busy?
A better question is:
Is the GPU being used efficiently while keeping inference healthy?
For example:
GPU: 95%
Queue: Low
Latency: Stable
Throughput: High
That may be perfectly healthy.
But:
GPU: 95%
Queue: Growing
Latency: Increasing
Errors: Increasing
tells a very different story.
The value comes from correlating signals rather than looking at one metric in isolation.
3. Inference Metrics
Inference metrics tell us what the AI service is actually doing.
The most useful ones include:
Request rate
Inference latency
Time to first token
Tokens per second
Queue depth
Concurrent requests
Model errors
Timeouts
A request may pass through several stages:
Request
↓
Queue
↓
Model Processing
↓
First Token
↓
Response Generation
↓
Complete Response
That gives us several useful timings:
- Inference latency: total time to complete the request
- Time to first token: how long the user waits before seeing the first output
- Tokens per second: generation speed after inference begins
- Queue depth: how many requests are waiting for capacity
Queue depth is especially useful as an early warning signal.
Imagine:
09:00 → 2 waiting requests
09:05 → 18
09:10 → 64
09:15 → 140
Nothing has crashed.
But demand is arriving faster than the available inference capacity can handle.
That signal can also feed autoscaling:
Queue grows
↓
Scaling signal
↓
More inference capacity
↓
Queue decreases
This is where AI-specific metrics become operational signals, not just dashboard numbers.
4. Observe the Complete Request Path
A production AI application is rarely just a model.
A request might travel through:
User
↓
API Gateway
↓
AI Application
↓
Model Server
↓
Vector Database
↓
External Tool
↓
Response
If the request takes eight seconds, we need to know where those eight seconds were spent.
Without tracing:
Request duration: 8.2 seconds
With tracing:
API Gateway 40 ms
Application 70 ms
Vector Search 420 ms
Model Inference 6.4 sec
External Tool 950 ms
Now the bottleneck is much easier to identify.
Metrics, Logs, and Traces Work Together
Each observability signal answers a different question.
Metrics
Is something wrong?
Example:
Inference latency increased 40%.
Logs
What happened?
Example:
Model request timed out after 10 seconds.
Traces
Where did it happen?
Example:
Most of the delay occurred during vector retrieval.
Together:
Metric Alert
↓
Latency increased
↓
Trace investigation
↓
Vector search is slow
↓
Logs
↓
Database connection pool exhausted
That is much more useful than looking at disconnected dashboards.
A correlation ID also helps connect these signals.
{
"request_id": "req-a91f82",
"service": "model-server",
"model": "model-v2",
"latency_ms": 1840,
"status": "success"
}
Now the same request can be followed through logs and traces across multiple services.
Avoid logging:
API keys
Access tokens
Passwords
Sensitive prompts
Private customer data
Confidential model responses
Observability should improve visibility without becoming a security risk.
A Practical AI Observability Stack
A simple cloud-native setup could look like:
Kubernetes
│
├── Application Metrics
├── GPU Metrics
├── Logs
└── Traces
│
▼
OpenTelemetry / Exporters
│
├── Prometheus
├── Log Backend
└── Trace Backend
│
▼
Grafana
Dashboards
Alerts
The tools may differ between organizations.
The pattern is what matters:
Collect
↓
Correlate
↓
Visualize
↓
Alert
↓
Investigate
Build Dashboards Around Questions
A dashboard with dozens of graphs can still be difficult to use.
Instead, build dashboards around operational questions.
Is the platform healthy?
Monitor:
Available replicas
Pod restarts
Node health
Request success rate
Are GPUs healthy?
Monitor:
GPU utilization
GPU memory
Temperature
Power
Device health
Is inference healthy?
Monitor:
Request latency
Time to first token
Tokens per second
Queue depth
Concurrent requests
Where are failures happening?
Monitor:
Errors by model
Errors by provider
Timeouts
Retries
Failed requests
That gives engineers somewhere useful to start during an incident.
Alert on User Impact
Not every metric needs an alert.
For example, 90% GPU utilization does not automatically mean there is a problem.
If throughput is high and latency is stable, the system may simply be using its resources efficiently.
A more useful alert might be:
Queue depth increasing
AND
Inference latency increasing
That points to something users are actually experiencing.
The best alerts are actionable.
Otherwise, teams eventually start ignoring them.
Tenant Visibility Matters
Multi-tenant AI platforms also need visibility by tenant.
Suppose the overall error rate reaches 15%.
That sounds serious.
But the breakdown might be:
Tenant A: 1%
Tenant B: 2%
Tenant C: 78%
Tenant D: 1%
Now the problem looks isolated rather than platform-wide.
This matters when tenants use different:
Model providers
API keys
Quotas
Tools
Workloads
Tenant-level visibility also helps with rate limiting, failure isolation, and capacity planning.
Observability Also Leads Into Cost
GPU utilization is not only a performance metric.
It is also a cost signal.
Consider:
Cluster A
GPU utilization: 82%
Cluster B
GPU utilization: 19%
If both clusters use similar hardware, Cluster B deserves investigation.
Maybe that spare capacity is intentional.
Or maybe the organization is paying for GPUs that spend most of their time idle.
The same applies to:
GPU hours
Tokens generated
Requests served
Model usage
These signals begin to connect infrastructure behavior with spend.
That leads directly into the next part of this series: FinOps for AI.
Practical AI Observability Checklist
Before running an AI workload in production, make sure you can answer:
- Are Kubernetes workloads healthy?
- Which GPUs are being used?
- How much GPU memory is consumed?
- What is the inference latency?
- What is the time to first token?
- How many requests are waiting?
- Can one request be traced across services?
- Are alerts tied to meaningful user impact?
If several of these questions cannot be answered quickly, there is probably an observability gap.
What's Next?
Observability tells us how the infrastructure behaves.
It also reveals something else:
How efficiently are we using the infrastructure we are paying for?
GPU hours, token usage, inference volume, idle capacity, and model selection all affect the economics of an AI platform.
In Part 4, we will look at:
FinOps for AI: Understanding GPU, Token, and Inference Costs
We will break down where AI infrastructure costs come from, why GPU utilization matters financially, and what cloud teams can measure to avoid unnecessary spend.
Final Thoughts
For AI infrastructure, CPU and memory are still important.
They are simply no longer enough.
A production AI platform needs visibility across:
Kubernetes
↓
GPU Infrastructure
↓
Model Serving
↓
Inference
↓
Dependencies
↓
User Experience
The real value comes from connecting those layers.
Instead of asking:
Why does the AI feel slow?
we want to be able to say:
Queue depth increased because the inference workers reached GPU capacity, which pushed time to first token above our target.
That is the difference between simply monitoring infrastructure and actually understanding the system.
Thanks for Reading
This article is Part 3 of my AI Infrastructure for Cloud Engineers series:
- Why Kubernetes Is Becoming the Operating System for AI Infrastructure
- Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving
- Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory
- Coming next: FinOps for AI: Understanding GPU, Token, and Inference Costs
- Building a Production AI Platform: Kubernetes, GitOps, IaC, Security, and Observability
I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, observability, and the engineering behind production AI systems.
LinkedIn: Connect with me on LinkedIn
If you're operating AI workloads in production, which signal has been most useful for you: GPU utilization, inference latency, time to first token, queue depth, or something else?
