S3 vs NFS for Kubernetes: Where Each One Actually Wins

S3 and NFS solve different Kubernetes storage problems. Use NFS or a CSI file driver when pods need shared, POSIX-style read-write-many file access. Use S3-compatible object storage when apps call the S3 API for immutable blobs, backups, and data lakes. RustFS gives you that S3 layer self-hosted under Apache 2.0.

Key Stats

What is NFS storage in Kubernetes?

NFS is a network filesystem that lets multiple pods read and write the same files across the network. In Kubernetes you usually consume it through the CSI NFS driver, whose plugin name is nfs.csi.k8s.io and which sig-storage ships as GA for Kubernetes 1.21 and later. The driver needs an existing NFSv3 or NFSv4 server and provisions Persistent Volumes by creating one subdirectory per PVC. NFS is the canonical ReadWriteMany filesystem: many pods on many nodes open the same file at once, which a block volume such as EBS or a local PV cannot do. We reach for it with shared model checkpoints, Jupyter home directories, and CMS uploads. The catch is that throughput tracks your filer, so a single NetApp or TrueNAS appliance becomes both the ceiling and the single point of failure. NFS itself has no built-in replication or tiering, so that lives on the backing storage.

What is S3-compatible object storage in Kubernetes?

S3-compatible object storage presents data as immutable blobs addressed by a key over an HTTP API, not as a POSIX filesystem. In Kubernetes there is no native "s3" volume type, so pods reach it through the application's AWS SDK or an S3 client such as aws s3 or mc, with credentials normally injected from a Secret. RustFS is an Apache 2.0, S3-compatible server built in Rust; its Feature & Status table lists S3 Core, Versioning, Object Lock (WORM), Lifecycle, and Distributed Mode all as Available. The mental model matters more than any benchmark: objects are addressed by key, versioned, and immutable until overwritten. There is no file handle, no in-place append, and no directory tree the kernel understands. For backups, artifacts, data lakes, and any app that already speaks the S3 API, this is the natural fit. We treat NFS and S3 as two separate tiers, not competitors.

The POSIX gap: why S3 doesn't mount like NFS

Object storage is not a filesystem, and that gap is the whole story. If a pod needs a path such as /data/shared/models/ckpt.bin with real POSIX semantics, NFS hands it over natively. S3 gives you a bucket and a key; to make it look like a directory tree you mount it through a FUSE layer such as s3fs-fuse. We have run s3fs-fuse in production, and its own README is blunt about the limits: random writes or appends rewrite the entire object, directory listings suffer network latency, there are no hard links and no atomic renames, and multiple clients mounting the same bucket get no coordination. s3fs-fuse preserves only a "large subset of POSIX" — mode, uid/gid, symlinks — not the full contract. So if your app does in-place edits, file locking, or expects rename() to be instant, a FUSE mount will bite you. Own it up front: S3 wins as an API, loses as a filesystem.

ReadWriteMany: where NFS still wins

ReadWriteMany, meaning many pods writing the same files at once, is NFS territory. Kubernetes access modes are ReadWriteOnce, ReadOnlyMany, and ReadWriteMany; a block volume or local PV is ReadWriteOnce by default, so a second node cannot attach it. NFS was built for concurrent multi-client access over the network, so the csi-driver-nfs driver happily serves 50 replicas of a web frontend from one share. We use NFS for Jupyter home directories and shared training caches where every pod must see the same file tree. The operational cost is real: you run and monitor an NFS server, and a file lock held by a crashed pod can stall others until the lock expires. If your requirement is "shared mutable files across nodes," deploy csi-driver-nfs and stop debating. S3 cannot do this without a FUSE shim, and that shim gives you no POSIX locking.

Immutable blobs, backups, and data lakes: where S3 wins

S3 pulls ahead the moment your data is immutable, append-only, or key-addressed. Backups are the obvious case: our nightly PostgreSQL dumps and CI artifacts land in RustFS as versioned objects, so a bad overwrite is one versionId away from recovery, not a restore-from-tape drama. Object storage also scales horizontally without a single filer ceiling; RustFS Distributed Mode is marked Available, so you grow capacity by adding nodes rather than buying a bigger appliance. Data lakes and Iceberg tables are S3-native: you write Parquet through the S3 API and let the catalog track partitions. RustFS lists S3 Tables (Iceberg REST) as Preview, so the Iceberg control plane is still maturing — we flag that honestly rather than pretend it is GA. For ML datasets, log archives, and any pipeline that streams objects, S3 is the lingua franca, and NFS would only add latency and inode headaches.

How do you choose between them on a real cluster?

Start from the access pattern, not the technology. Ask four questions. First, do pods need POSIX file APIs and in-place edits? If yes, NFS. Second, do many pods write the same files at once? If yes, NFS or a CSI file driver. Third, is the data immutable blobs, backups, or a data lake? If yes, S3. Fourth, does the app already call the S3 API? If yes, S3, and skip the filesystem debate. We draw the line at mutability: mutable-shared-files maps to NFS, immutable-keyed-objects maps to S3. RustFS slots into the S3 side as a self-hosted, Apache 2.0 drop-in you control, with no egress fees and no vendor lock. The anti-pattern is mounting object storage as a filesystem just because the URL is shorter; you pay the FUSE tax for zero benefit. Pick the primitive that matches the workload and stop there.

Running RustFS as your S3 tier in Kubernetes

You can stand up the S3 tier in minutes. RustFS ships Kubernetes Helm charts, marked Available in its Feature & Status, or you can run the container directly. The documented start command, verbatim from the project README, is:

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Default credentials are rustfsadmin / rustfsadmin — rotate these before production, since they ship as defaults. In Kubernetes we expose the S3 endpoint (port 9000) and the Web Console (9001) through a Service, inject the access and secret key from a Secret, and point pods at the endpoint with the AWS SDK or mc. RustFS is Apache 2.0 and built in Rust; a 3-node Distributed Mode deployment fits commodity hardware. Because it is S3-compatible, existing tooling — aws s3, rclone, the MinIO client — works unchanged. NFS stays a separate concern; RustFS complements it rather than replacing it.

FAQ

Can I mount S3 as a filesystem in Kubernetes?

Yes, via a FUSE layer such as s3fs-fuse (GPLv2), which mounts a bucket as a local path. But s3fs-fuse's own README warns of real limits: random writes rewrite the whole object, there are no hard links, no atomic renames, and no coordination between multiple clients mounting the same bucket. Treat it as a read-mostly or write-once mount, not a POSIX replacement.

Is the in-tree NFS volume deprecated in Kubernetes?

The nfs volume type still appears in the Kubernetes docs, but the maintained, recommended path is the CSI NFS driver (nfs.csi.k8s.io), which sig-storage ships as GA for Kubernetes 1.21 and later. It requires an existing NFSv3 or NFSv4 server and provisions PVs by creating a subdirectory per PVC. For new clusters, install csi-driver-nfs with Helm.

Which should I use for AI/ML training data?

Immutable, large datasets (Parquet, image shards, model checkpoints stored as objects) belong in S3-compatible object storage; RustFS serves them over the S3 API at scale. Shared, frequently-mutated small files or Jupyter home directories belong on NFS with ReadWriteMany. Mix both: object store for the corpus, NFS for scratch.

Can RustFS replace NFS?

No. RustFS is an S3-compatible object store and has no built-in POSIX or FUSE filesystem. Its Feature & Status table lists S3 Tables (Iceberg) and MinIO on-disk compatibility as Preview and does not list FUSE or POSIX. Use RustFS for the object tier and keep a CSI file driver such as csi-driver-nfs for shared files.

How do pods access RustFS?

Through the S3 API: inject the access and secret key from a Kubernetes Secret and point the AWS SDK or mc at the RustFS Service endpoint (port 9000). There is no native Kubernetes volume type, so apps must speak S3. If you must have a filesystem view, run s3fs-fuse as a sidecar or init container that mounts the bucket.