Welcome back to my blog series, "Into The Depths of Kubernetes," where we are peeling back the abstraction layers to explore the architectural patterns, runtime mechanics, and production realities that truly matter for software and platform engineers. Whether you are scaling microservices, optimizing infrastructure costs, or hardening cluster security, this series is designed to give you actionable insights and deep technical clarity. We are kicking things off with a fundamental challenge every growing organization faces: Multi-Tenancy in Kubernetes. In this inaugural post, we will dive into isolating workloads, managing shared cluster resources, enforcing strict security boundary conditions with Namespaces and Network Policies, and balancing cost efficiency against robust tenant isolation.
The Data Plane & Storage Subsystem: eBPF Firewalls, CSI Scoping, and Cryptographic Isolation
Multi-tenancy in Kubernetes does not end when the API server rejects an unauthorized request.
In the first two installments of our Into The Depths of Kubernetes multi-tenancy series, we moved beyond basic logical namespace isolation to dissect the underlying control-plane and node-level bottlenecks. In Part 1 exploded the Kubernetes control plane, examining API request concurrency, how API Priority and Fairness (APF) prevents request saturation, CRD name-collision risks, and the case for virtual control planes like vCluster. Part 2 took us a layer deeper into the Linux kernel, demystifying hard workload sandboxing through cgroups v1 vs. v2 hierarchies, CPU throttling traps, OOM killer mechanics, and secure runtime boundaries like gVisor and Kata Containers. Together, these posts laid the architectural and kernel-level foundations required to truly understand high-density, multi-tenant cluster design.
That is where things get considerably more interesting.
A compromised container does not necessarily care about your carefully designed RBAC policies. It cares about what it can reach over the network, what it can discover through DNS, what interfaces it can access, and what data it can read from mounted storage. If Tenant A can reach Tenant B's backend service, or if a compromised pod can escape its volume boundary and read another tenant's persistent data, the isolation model has already failed regardless of how well the Kubernetes API is protected.
This final part of the series moves beneath the Kubernetes API and looks at the infrastructure where tenant workloads actually exchange packets and persist data.
The objective is simple: make tenant boundaries hold even when the workload itself is no longer trustworthy.
The Data Plane Is Where Multi-Tenancy Gets Real
There is a common mistake in multi-tenant Kubernetes design: treating namespaces as if they were security boundaries by themselves.
They are not.
A namespace gives Kubernetes a logical scope for resources. RBAC can control who can interact with that namespace. ResourceQuota can control how much a tenant consumes. Admission policies can restrict what workloads are allowed to create. But none of those mechanisms automatically prevents a running workload from communicating with another workload or accessing data exposed through a poorly isolated storage subsystem.
Consider a cluster hosting three tenants:
Kubernetes Cluster
│
┌────────────────┼────────────────┐
│ │ │
Tenant A Tenant B Tenant C
│ │ │
Web/API Pods Web/API Pods Web/API Pods
│ │ │
└────────── Network ──────────────┘
│
Storage
│
Shared Infrastructure
If Tenant A compromises one of its containers, the attack does not stop at the container boundary. The attacker will typically start looking sideways.
- Can I resolve another tenant's service?
- Can I connect to its database?
- Can I access an internal metrics endpoint?
- Can I reach the Kubernetes API?
- Can I discover mounted files?
- Can I access another tenant's persistent volume?
The answer to those questions is determined primarily by the data plane, not the control plane.
That is why serious multi-tenancy requires two independent assumptions:
Network isolation must survive a compromised workload.
and
Storage isolation must survive access to the underlying storage infrastructure.
The first requires strong packet-level enforcement. The second requires storage boundaries that do not depend solely on directory permissions or mount configuration.
This is where eBPF networking and cryptographic storage isolation become particularly useful.
Bypassing IPTables: The eBPF Networking Revolution
For years, Kubernetes networking security has been heavily associated with iptables.
The basic model is straightforward. A Kubernetes NetworkPolicy is translated by the networking implementation into rules that eventually influence packet filtering in the Linux networking stack. As workloads and policies grow, however, the amount of state that the node has to maintain can become significant.
The problem is not that iptables suddenly becomes insecure. The problem is scale and enforcement efficiency.
Traditional iptables processing is fundamentally rule-oriented. Packets encounter chains containing rules that must be evaluated until a matching condition determines the result. In a small cluster this is rarely something you lose sleep over. In a large multi-tenant environment with thousands of workloads, services, endpoints, and policy relationships, the number of rules and the amount of churn can become substantial.
Every workload event can potentially change the effective networking state. A new pod appears. An endpoint changes. A service is updated. A NetworkPolicy changes. A node receives more workloads. The networking layer must continuously reconcile these changes.
With enough tenants and enough policy relationships, this becomes another scalability problem in the node data plane.
eBPF Changes Where the Enforcement Happens
eBPF approaches the problem from a different direction.
Instead of relying exclusively on large rule chains, an eBPF-based networking implementation can load programs directly into the Linux kernel and attach them at strategic points in the networking path. The program can inspect packet metadata, identities, sockets, ports, protocols, and other context and make an enforcement decision without requiring every packet to walk through a massive iptables rule set.
This is one reason projects such as Cilium have become important in Kubernetes networking.
A simplified conceptual path looks like this:
Traditional:
Pod
│
▼
Linux Networking Stack
│
▼
iptables chains
│
├── Rule 1
├── Rule 2
├── Rule 3
├── Rule 4
├── ...
└── Rule N
│
▼
Destination
eBPF-oriented:
Pod
│
▼
eBPF Hook
│
├── Identity lookup
├── Policy lookup
├── Security decision
│
├── DROP
└── ALLOW
│
▼
Destination
The important part is not simply that eBPF is "faster."
The more interesting property for multi-tenancy is that the networking layer can reason about workload identity and enforce policy closer to the point where traffic enters the kernel networking path.
That gives the platform considerably more control over lateral movement.
Instead of thinking only in terms of:
Source IP → Destination IP → Port
modern Kubernetes networking can reason more naturally about:
Tenant
↓
Namespace
↓
Workload identity
↓
Service identity
↓
Policy
That shift is extremely useful in environments where IP addresses are ephemeral and workloads are continuously recreated.
Identity Matters More Than IP Addresses
IP-based security rules are awkward in Kubernetes because IP addresses are temporary implementation details.
A pod disappears. Another pod replaces it. The replacement receives a different address. The security relationship, however, has not changed.
If the security model is "Tenant A's frontend can talk to Tenant A's backend," you do not actually care whether the backend currently has 10.42.4.18 or 10.42.7.31. You care about who owns that workload and what role it plays. This is where identity-aware networking becomes valuable.
A policy can conceptually express:
Allow:
tenant-a/frontend
↓
tenant-a/backend:8080
while denying:
tenant-a/frontend
X
tenant-b/backend:8080
The distinction becomes particularly important when tenants share nodes. Two pods belonging to completely different customers may be running side by side on the same Linux host, and the network policy needs to maintain that boundary regardless of their physical placement.
That is the data-plane version of multi-tenancy.
Default-Deny-All Is the Starting Point
If tenants are supposed to be isolated, the safest starting assumption is that nothing should communicate unless it has been explicitly permitted.
A default-deny policy is therefore one of the most important controls in a multi-tenant cluster.
A basic Kubernetes NetworkPolicy can establish this boundary:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: tenant-a
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
This does something intentionally boring.
It says:
Every pod in
tenant-astarts with no allowed ingress or egress traffic.
That is exactly what we want from a zero-trust starting point. But there is an immediate problem. A completely isolated namespace cannot do much useful work. Applications still need DNS. They may need access to an internal database. They may need to communicate with an ingress gateway. They may need access to a monitoring endpoint.
The solution is not to abandon default deny. The solution is to add small, explicit exceptions.
Allowing DNS Without Opening the Door
DNS is one of the easiest places to accidentally weaken a supposedly strict NetworkPolicy.
Kubernetes workloads commonly use CoreDNS through the cluster DNS service. If DNS traffic is blocked, applications begin failing in ways that look completely unrelated to networking policy.
A production policy therefore normally needs an explicit DNS exception.
For example:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: tenant-a
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
The important detail is that DNS is being allowed to the DNS workload, rather than allowing unrestricted traffic to the entire kube-system namespace.
That distinction matters.
This:
tenant-a → kube-system
is broad.
This:
tenant-a → CoreDNS:53
is narrow. The principle should always be:
Allow the dependency, not the infrastructure namespace.
The same model applies to other shared services. If tenants need access to a central observability endpoint, expose only the required destination and port. If they need an ingress gateway, allow the gateway identity rather than the entire ingress namespace.
Building a Tenant Network Boundary
Once DNS is handled, the tenant network can become deliberately restrictive.
A simplified architecture might look like this:
Cluster
│
┌───────────────┼────────────────┐
│ │ │
Tenant A Tenant B Tenant C
│ │ │
┌────┴────┐ ┌────┴────┐ ┌────┴────┐
│ Frontend│ │ Frontend│ │ Frontend│
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
Backend A Backend B Backend C
│ │ │
X X X
│ │ │
└────── Cross-Tenant DENY ───────┘
Within the tenant, you can then explicitly define allowed paths:
Frontend
│
├── DNS
│
└── Backend:8080
Backend
│
├── DNS
│
└── Database:5432
Everything else remains denied. This is much easier to reason about than trying to enumerate every possible unwanted connection. The security model becomes:
Default = DENY
Explicitly required dependency = ALLOW
That is a much stronger foundation for multi-tenancy.
The Storage Loophole
Network isolation solves only half of the data-plane problem. The other half is storage. And storage is where many seemingly secure Kubernetes environments have a surprisingly large hole.
Imagine two tenants:
Tenant A → PVC-A
Tenant B → PVC-B
At the Kubernetes API level, these appear completely independent. But eventually those PVCs map to something underneath:
PVC-A ──► PV-A ──► Storage Backend
PVC-B ──► PV-B ──► Storage Backend
The Kubernetes abstraction can make those volumes look isolated while the underlying implementation may not be.
This becomes especially dangerous when operators use hostPath, shared NFS exports, custom CSI drivers, or storage backends that were not designed with hostile tenants in mind.
CSI Is a Security Boundary, Not Just a Storage Plugin
The Container Storage Interface provides a standard mechanism through which Kubernetes interacts with storage systems.
That standardization is useful, but CSI itself does not magically make storage multi-tenant safe. The security depends on how the driver provisions, mounts, and exposes the underlying storage.
Consider a simplistic hostPath-style arrangement:
Node
│
└── /data
├── tenant-a/
│ ├── database.db
│ └── secrets/
│
└── tenant-b/
├── database.db
└── secrets/
A pod might receive:
/data/tenant-a
as its mounted filesystem. The assumption is that the container can only see that directory. But if the underlying hostPath configuration, privileged permissions, mount propagation, or driver implementation is flawed, that boundary can become surprisingly weak.
A compromised workload should never be trusted simply because Kubernetes mounted a particular directory into the container.
Directory Traversal Is Only One Failure Mode
The more serious problem is that storage isolation can fail at several layers.
You can have:
Container filesystem isolation
↓
Kubernetes volume isolation
↓
CSI driver isolation
↓
Node filesystem isolation
↓
Storage backend isolation
↓
Physical/cloud storage
A weakness at any layer can undermine the layers above it. A shared NFS server provides another example. Suppose multiple tenants receive directories from the same export:
NFS Export
│
├── /tenant-a
├── /tenant-b
└── /tenant-c
If permissions, UID/GID mapping, root squashing, export configuration, or CSI mount behavior is incorrectly configured, a compromised workload may be able to interact with storage that belongs to another tenant.
The critical point is that filesystem permissions are not the same thing as cryptographic isolation.
If two tenants ultimately depend on the same encryption key, a compromised storage credential, snapshot, backup, or underlying storage access path may still expose both datasets.
Zero-Trust Storage: Encryption Per Tenant
This leads to a stronger model.
Instead of asking:
"Can Tenant A access Tenant B's directory?"
ask:
"Even if Tenant A somehow obtains the underlying storage bytes, can it decrypt Tenant B's data?"
That is a much stronger security boundary.
The architecture becomes:
Kubernetes
│
Tenant Namespace
│
PVC
│
CSI
│
┌─────────┴─────────┐
│ │
Volume Provisioning Key Request
│ │
▼ ▼
Cloud Block Storage KMS
│
┌─────────┴─────────┐
│ │
Tenant-A Key Tenant-B Key
The critical property is that the storage encryption key becomes part of the tenant isolation model. Tenant A does not simply receive a different directory. Tenant A receives a volume encrypted under a key that Tenant B cannot use.
CSI + KMS: The Architecture
A production implementation can integrate a CSI driver with a Key Management Service.
Depending on the environment, the KMS might be:
- AWS KMS
- HashiCorp Vault
- Azure Key Vault
- Google Cloud KMS
- Another enterprise key-management platform
The workflow looks roughly like this:
1. Tenant creates PVC
│
▼
2. Kubernetes requests volume
│
▼
3. CSI driver provisions storage
│
▼
4. CSI requests encryption material
│
▼
5. KMS authorizes tenant-specific key
│
▼
6. Volume is encrypted
│
▼
7. Pod receives mounted volume
The important part is step five. The KMS authorization layer should understand which tenant is requesting the key.
For example:
Tenant A
│
└──► kms/tenant-a-key
Tenant B
│
└──► kms/tenant-b-key
Tenant A should not have permission to request or unwrap Tenant B's encryption key. This creates a security boundary outside Kubernetes itself.
Why Encryption Changes the Threat Model
Consider an attacker who somehow obtains the raw storage device or snapshot.
Without encryption:
Attacker
│
▼
Storage Snapshot
│
▼
Read Database Files
With tenant-specific encryption:
Attacker
│
▼
Storage Snapshot
│
▼
Encrypted Bytes
│
X
│
Missing Tenant-B Key
The data is still physically present. The attacker simply cannot turn those bytes into meaningful information without the cryptographic material required to decrypt them.
That distinction becomes extremely important for cloud environments, snapshots, backups, storage migrations, and infrastructure operators who may have access to the storage layer but should not automatically have access to tenant data.
KMS Policy Becomes Part of Kubernetes Security
This is where the architecture becomes more interesting. You now have security controls distributed across multiple layers:
Kubernetes RBAC
│
▼
Namespace Boundary
│
▼
NetworkPolicy / eBPF
│
▼
CSI Provisioning
│
▼
KMS Authorization
│
▼
Encrypted Storage
The tenant's identity must therefore propagate through the infrastructure correctly. A Kubernetes ServiceAccount might identify the workload.
The CSI driver provisions the volume. The cloud identity layer determines which KMS operation the workload or infrastructure component can perform.
The KMS policy then determines which encryption key can actually be used. This produces something much closer to a true zero-trust storage model.
A Practical Multi-Tenant Storage Model
For a serious multi-tenant platform, I would think about storage in terms of three independent boundaries.
Boundary 1: Provisioning
A tenant should only be able to request storage through approved StorageClasses and CSI drivers. Avoid giving tenants arbitrary access to hostPath volumes. Avoid allowing workloads to manipulate mounts. Avoid exposing privileged storage operations to ordinary application pods.
Boundary 2: Physical Isolation
Where practical, separate tenants at the storage backend.
For example:
Tenant A → Volume Group A
Tenant B → Volume Group B
Tenant C → Volume Group C
This does not necessarily mean dedicated physical hardware. It can mean separate logical storage resources, access policies, projects, accounts, or backend namespaces depending on the storage platform.
The goal is to reduce the blast radius of a storage-layer compromise.
Boundary 3: Cryptographic Isolation
Finally:
Tenant A → Key A
Tenant B → Key B
Tenant C → Key C
Even if the physical storage boundary is bypassed, the cryptographic boundary remains.
This is the strongest layer because it does not depend on directory names, Linux permissions, or Kubernetes object ownership.
The Real Architecture
Put everything together and the multi-tenant data plane begins to look like this:
Kubernetes Cluster
│
┌─────────────────┴─────────────────┐
│ │
CONTROL PLANE DATA PLANE
│ │
RBAC / Admission eBPF / CNI
│ │
Namespace Policy Default Deny
│ │
│ Explicit Allow
│ │
│ Tenant Identity
│ │
│ ▼
│ Network Boundary
│
└──────────────────────┬────────────────────
│
CSI
│
Storage Provisioning
│
▼
KMS
│
┌────────────────┼────────────────┐
│ │ │
Key A Key B Key C
│ │ │
▼ ▼ ▼
Tenant A Tenant B Tenant C
Volume Volume Volume
There is no single magical Kubernetes feature providing this isolation. It is the composition of several independent security boundaries. That is the important lesson. ## What Happens When One Layer Fails?
This is perhaps the most useful way to evaluate a multi-tenant platform. Assume Tenant A's application is compromised.
Scenario 1: Network compromise
The attacker attempts:
Tenant A → Tenant B
The eBPF or NetworkPolicy layer rejects the traffic.
Scenario 2: Service discovery abuse
The attacker attempts to discover internal services. DNS may still work, because DNS is explicitly required. But DNS access does not automatically provide network access to the discovered service.
That distinction is important.
DNS lookup = ALLOWED
Connection to Tenant B backend = DENIED
Scenario 3: Storage compromise
The attacker somehow gains access to another tenant's storage bytes. The volume data is encrypted. Without the appropriate KMS key, the raw bytes are useless.
Scenario 4: Node compromise
This is where things become significantly more difficult.
If an attacker obtains full root access to a Kubernetes worker node, many workload-level security assumptions are already under pressure. This is why node hardening, confidential computing where appropriate, restricted privileges, kernel security, runtime isolation, and careful separation of highly sensitive workloads remain important.
Multi-tenancy is therefore not about creating an absolute wall.
It is about creating multiple independent barriers so that compromising one layer does not automatically compromise everything above or below it.
The Platform Engineer's Final Checklist
Before calling a Kubernetes cluster genuinely multi-tenant, I would want to answer these questions clearly.
Networking
- Is every tenant namespace default-deny?
- Are cross-namespace connections explicitly controlled?
- Is DNS allowed only to the required CoreDNS endpoints?
- Are policies identity-aware where possible?
- Can a compromised workload reach node-local services?
- Can workloads bypass the CNI?
- Are privileged pods restricted?
Storage
- Are hostPath volumes prohibited for ordinary tenants?
- Is the CSI driver trusted and correctly configured?
- Can a tenant access another tenant's mount?
- Is shared NFS configured with strong isolation?
- Are volume snapshots isolated?
- Are backups protected by the same tenant boundary?
- Can storage credentials be reused across tenants?
Cryptography
- Does each tenant have an independent encryption boundary?
- Can Tenant A request Tenant B's key?
- Are KMS permissions tied to workload or tenant identity?
- Are encryption keys rotated?
- Are snapshots encrypted?
- Are backup copies encrypted?
- What happens when a tenant is deleted?
That final question is often forgotten. Deleting a namespace does not necessarily mean deleting its data. A mature platform must define what happens to:
PVC
↓
Volume
↓
Snapshot
↓
Backup
↓
Encryption Key
Tenant offboarding is therefore also a cryptographic lifecycle problem.
The Final Layer of Multi-Tenancy
Multi-tenancy is often introduced as a Kubernetes namespace problem.
It is not.
Namespaces provide a useful logical boundary, but serious isolation requires the platform to enforce boundaries across the entire workload lifecycle.
The control plane decides who is allowed to do what. The scheduler decides where workloads can run. The network layer decides who can communicate with whom. The storage layer decides who can access which data.
And the cryptographic layer decides whether that data is useful even when the underlying storage boundary has been breached.
That is why eBPF, CSI architecture, and KMS integration matter so much in advanced Kubernetes environments. They push tenant isolation below the Kubernetes object model and into the infrastructure that actually carries packets and stores bytes.
A useful way to think about the entire series is:
Multi-Tenant Kubernetes
│
┌──────────────┼──────────────┐
│ │ │
Control Plane Workloads Data Plane
│ │ │
RBAC Runtime eBPF/CNI
Admission Security NetworkPolicy
Quotas Scheduling │
│ │ │
└──────────────┼──────────────┘
│
CSI
│
KMS
│
Cryptographic Boundary
And that is ultimately the difference between running multiple tenants on Kubernetes and building Kubernetes as a multi-tenant platform.
The second requires you to assume that eventually, something will fail.
A container will be compromised. A credential will leak. A policy will be misconfigured. A node will behave unexpectedly. A storage snapshot will escape the normal workflow.
The question is not whether you can prevent every possible failure. The question is whether one failure gives an attacker the keys to everything else.
A well-designed multi-tenant Kubernetes platform says no.
Network boundaries stop lateral movement. CSI boundaries restrict storage access. KMS-backed encryption protects data even when lower-level assumptions fail. And eBPF gives the data plane a way to enforce those decisions efficiently as the number of workloads and tenants grows.
That is where multi-tenancy becomes more than namespace isolation. It becomes an exercise in containing failure, limiting blast radius, and making every layer assume that the layer beneath it might eventually be compromised.
That is the real depth of Kubernetes multi-tenancy.
