Almost every "Jellyfin cannot handle four people at once" problem is a transcoding problem, not a bandwidth or server power problem. If your CPU pins at 100% with three or four viewers, the server is re-encoding video in software because hardware acceleration is off, the GPU is not visible inside the container, a subtitle track is being burned in, or a remote client asked for a lower bitrate than the file. Fix the transcode path and the same hardware that choked on two streams will usually carry four or more. The work is configuration, not new hardware, and you can verify each fix from the Jellyfin dashboard in minutes.
TL;DR by household setup:
- One 4K TV in the living room, everything else on phones (a couple with an Apple TV and two iPhones): enable hardware acceleration and keep a second 1080p copy of your most watched films, because the phones will never direct play a 4K HDR remux.
- Two parents plus two teenagers streaming to different rooms at the same time (a four person house on one server): set per user bitrate limits and enable hardware decoding and encoding first, since concurrent software transcodes are what saturates the CPU.
- Grandparents watching remotely over a home upload link (one remote viewer, one local): cap the remote user's bitrate in the user policy and accept a deliberate transcode, because unlimited remote playback of a 40 Mbps file will stall on most residential upload speeds.
- A mini PC or NAS with Intel integrated graphics (an N100 or a recent Synology or Beelink box): pass /dev/dri into the container and turn on Intel QuickSync, because that single change moves the encode off the CPU entirely.
- An old desktop with a dedicated NVIDIA card (a repurposed tower in the basement): use NVENC and check the concurrent session limit on consumer cards, since driver level caps, not the GPU itself, decide how many streams you get.
- A family that mostly watches on browsers and smart TV apps (mixed Chromecast, Fire TV and laptop viewing): standardise on H.264 or widely supported HEVC with AAC audio, because container and audio mismatches trigger transcodes even when the video codec is fine.
The central tradeoff is simple: every gigabyte of disk you spend on client friendly copies of your media is CPU and GPU work you never have to do while four people are watching.
Table of contents
- Why does Jellyfin hit 100% CPU when only three or four people are watching?
- How do you tell whether a Jellyfin stream is direct playing, remuxing or transcoding?
- Anti-pattern 1: hardware acceleration left off, or switched on but never verified
- Which transcoding path fits your server: Intel QuickSync, NVIDIA NVENC, AMD VAAPI or software?
- Why does the GPU never reach your Jellyfin Docker container?
- Anti-pattern 2: keeping one 4K HDR file as the only copy for every screen in the house
- Why do subtitles turn a smooth Jellyfin stream into a full video re-encode?
- Anti-pattern 3: leaving remote family accounts with no bitrate ceiling
- Anti-pattern 4: pointing the Jellyfin transcode cache at the wrong disk
- Which audio codecs and containers quietly force your family's clients to transcode?
Why does Jellyfin hit 100% CPU when only three or four people are watching?
Serving a file costs almost nothing. Re-encoding it costs everything. When Jellyfin direct plays, the server reads bytes off disk and pushes them to the client, which is disk and network work that a ten year old CPU handles for a dozen viewers at once. When Jellyfin transcodes in software, it decodes every frame and re-encodes it with libx264 or libx265, and that single stream will use every core you give it. Three concurrent software transcodes on a four core CPU is not a load problem, it is an impossible request.
What actually consumes the CPU: the software encoder, not the number of sessions. One 4K HEVC to 1080p H.264 software transcode can occupy a modern desktop CPU on its own, which is why the second viewer arrives to buffering.
Why it looks like a bandwidth problem: transcoding cannot keep ahead of playback, so the client empties its buffer and pauses, exactly the symptom a slow network produces.
Why it starts at three streams and not one: the first transcode uses spare headroom, the second consumes it, and the third queues behind both.
Why your hardware is probably fine: an Intel CPU with QuickSync, or any GPU with a dedicated encoder block, offloads the work to fixed function silicon and leaves the general purpose cores idle.
Why the trigger is often invisible: a subtitle track, an audio codec or a client bitrate cap flips a session from direct play to transcode without anyone changing a setting.
Run top or htop during playback. If ffmpeg sits at the top, you have found your answer.
How do you tell whether a Jellyfin stream is direct playing, remuxing or transcoding?
Stop guessing and read the session. Open Dashboard while someone is watching: each active device shows the playing method under the title, along with source and output bitrates. On the client side, the Jellyfin web player exposes the same data through the Playback Info entry in the player menu. That one screen tells you whether the server is copying bytes or rebuilding frames, and Jellyfin also prints the trigger, in plain language, such as "The audio codec is not supported".
| Playing method | What the server does | What it costs you |
|---|---|---|
| Direct Play | Sends the original file untouched, container and all streams intact | Disk reads and network only, negligible CPU, scales to many viewers |
| Direct Stream (remux) | Repackages the same video into another container, video bitstream copied | Low CPU, a few percent, output bitrate stays equal to the source |
| Audio transcode only | Copies the video, re-encodes the audio track to AAC or similar | Small and constant CPU cost, safe to run several at once |
| Video transcode | Decodes and re-encodes every frame at a new resolution or bitrate | The expensive case, and the one that pins cores when several run together |
Two habits make this permanent. First, check the source and output bitrate pair in the session card: identical numbers mean no video re-encode, a drop from 38 Mbps to 8 Mbps means one is happening. Second, keep jellyfin.log open in a second window during a test playback, because the transcode reason is recorded there for clients that hide the overlay.
Anti-pattern 1: hardware acceleration left off, or switched on but never verified
The default install has hardware acceleration set to None. Every transcode is software until you change it, so a brand new Jellyfin server on capable hardware behaves exactly like a server with no GPU at all. The second mistake is worse, because it looks solved: you pick an acceleration mode in Dashboard, Playback, Transcoding, save, and never confirm that a single frame was actually decoded on the GPU.
Turn it on in the right place: Dashboard, Playback, Transcoding, then choose the mode that matches your chip, not the first entry in the list.
Tick the decode codecs you actually own: enabling H.264 and HEVC decoding is separate from choosing the mode, and leaving those boxes empty means the GPU decodes nothing.
Enable hardware encoding as well as decoding: decode only acceleration still leaves libx264 on the CPU, which is the expensive half of the job.
Verify with the transcode log: open a transcoding session, then read the newest file in /config/log/, where a working setup shows the hardware device being initialised rather than a plain software encoder line.
Verify with the server itself: run nvidia-smi on an NVIDIA host or vainfo inside the container on an Intel or AMD host, during playback, and check the encoder is busy.
Where you run Jellyfin decides which of these options exists at all. A home NAS, a self managed VPS, a mini PC under the TV and Yundera are all viable hosts. Yundera is a managed Personal Cloud Server, built on CasaOS, that runs self-hosted apps as Docker containers on a server dedicated to the user. Rented virtual machines frequently expose no GPU, so confirm before you buy.
Which transcoding path fits your server: Intel QuickSync, NVIDIA NVENC, AMD VAAPI or software?
Pick the path your silicon already has, not the one with the best reputation. Jellyfin lists the modes in one dropdown, and choosing a mode your hardware does not implement produces failed playback rather than a fallback you will notice. Check the chip first, then set the matching entry.
| Path | Fits which server | What to check before you commit |
|---|---|---|
| Intel QuickSync (QSV) | Mini PCs, most NAS units and any Intel CPU with integrated graphics enabled in the BIOS |
/dev/dri/renderD128 must exist on the host, and the iGPU must not be disabled when a discrete card is installed |
| NVIDIA NVENC | A tower with a GeForce or Quadro card already in it | Consumer drivers have historically capped concurrent encode sessions, so read NVIDIA's current encode support matrix before planning four streams |
| AMD VAAPI or AMF | AMD APUs and Radeon cards on Linux hosts | Mesa driver version and container packaging decide what works, and HEVC support varies more than on Intel |
| Apple VideoToolbox | A Mac mini running Jellyfin natively | Not available to a Docker container, so this rules out the containerised setup entirely |
Software (libx264) |
Any machine, as the fallback when no encoder block exists | Predictable quality, unpredictable capacity, and the reason most family servers stall at two viewers |
Hosting choice constrains this list. A NAS, a home mini PC, a self managed VPS and Yundera are all places Jellyfin runs. Yundera is a managed Personal Cloud Server, built on CasaOS, that runs self-hosted apps as Docker containers on a server dedicated to the user, installed from an app store in one click. Whichever you choose, confirm GPU access before you build a library around 4K files.
Why does the GPU never reach your Jellyfin Docker container?
A container sees only the devices you hand it. The host can have a perfectly working iGPU, vainfo can succeed on the host, and the container still has nothing to accelerate with. This is the single most common reason a correctly configured Jellyfin setting produces software transcodes anyway.
No device mapping in the compose file: Intel and AMD hosts need devices: - /dev/dri:/dev/dri in the service definition, and adding it requires recreating the container, not restarting it.
Wrong group membership inside the container: the render node is owned by a group whose numeric GID differs between distributions, so read it with ls -l /dev/dri on the host and pass the same value through group_add in compose.
NVIDIA cards need the toolkit, not a device line: install the NVIDIA Container Toolkit on the host and reserve the GPU in compose, or set NVIDIA_VISIBLE_DEVICES=all and NVIDIA_DRIVER_CAPABILITIES=all for the container.
A hypervisor layer in the way: Jellyfin inside a Proxmox LXC or a virtual machine only sees the GPU if the host was configured to pass it through first, and an unprivileged LXC needs explicit cgroup device rules.
A driver mismatch between host and image: the container ships its own userspace drivers, so an old host kernel or a very new GPU can leave you with a device node that opens but exposes no encode profiles.
Verify with one command: docker exec -it jellyfin ls -l /dev/dri. If that returns nothing, no Jellyfin setting will help. If it lists renderD128, run vainfo in the same container and read which encode profiles are actually advertised.
Anti-pattern 2: keeping one 4K HDR file as the only copy for every screen in the house
One pristine UHD remux looks like good curation. It is actually a transcode generator. A UHD Blu-ray source can carry video bitrates near 100 Mbps, and no phone, tablet or older smart TV in the house will direct play that. Every screen that is not the living room TV forces a downscale, and HDR makes it worse, because converting HDR10 to SDR requires tone mapping on top of the resize.
Tone mapping is a second workload: Jellyfin does it on the GPU through OpenCL or vendor specific paths, and enabling hardware acceleration without enabling tone mapping leaves the colour conversion on the CPU or produces washed out, grey looking video.
Dolby Vision profile 5 has no usable fallback: clients that cannot decode it show green and purple frames, so those files always need re-encoding for the rest of the household.
Disk is the cheaper resource: a second 1080p H.264 copy costs a few gigabytes per film once, while transcoding the 4K original costs GPU time on every single play.
Jellyfin has a built in mechanism for this: put both files in the same movie folder and name them Film Name (2021) - 4K.mkv and Film Name (2021) - 1080p.mkv, and Jellyfin presents them as selectable versions of one title.
Target the copy at your real clients: H.264 in MP4 or MKV, 1080p, SDR, is the widest compatibility baseline across browsers, Chromecast, Fire TV and phones.
Do this for the twenty titles your family rewatches, not the whole library. That handles most concurrent playback without touching the archive quality originals.
Why do subtitles turn a smooth Jellyfin stream into a full video re-encode?
Because some subtitles are pictures, not text. Text subtitles can be sent to the client as a separate stream and drawn by the player. Image subtitles have to be painted onto the video frames themselves, which means decoding and re-encoding every frame. A family member turning on subtitles is therefore enough to convert a zero cost direct play into the most expensive session on the server, and it happens mid playback with no warning.
| Subtitle type | How Jellyfin delivers it | Cost to the server |
|---|---|---|
| SRT, external or embedded | Sent to the client as a separate text track | None, video keeps direct playing |
| WebVTT | Converted from text and streamed alongside the video | Negligible, a text conversion only |
| ASS or SSA with styling | Burned in when the client cannot render the styling | Full video re-encode for the whole session |
| PGS, from Blu-ray sources | Always burned in, since it is a bitmap format | Full video re-encode, the usual hidden cause |
| VobSub, from DVD sources | Always burned in, same bitmap problem | Full video re-encode |
Three fixes, in order of effort. Set the burn in policy in Dashboard, Playback, under the subtitle burn in option, so text formats are never burned in unnecessarily. Convert bitmap tracks to text once with OCR, or fetch SRT sidecars with the OpenSubtitles plugin and drop them beside the video file as Film Name (2021).en.srt. Then extract embedded text tracks to sidecars with ffmpeg -i input.mkv -map 0:s:0 output.srt, so Jellyfin never has to pull them out of the container during playback.
Anti-pattern 3: leaving remote family accounts with no bitrate ceiling
A grandparent's tablet on the other side of the country asks for the same file the living room TV gets. Nothing in Jellyfin stops it by default. The client requests the source, the request exceeds your home upload capacity, playback stalls, and the server keeps working anyway. Worse, the client often reacts by asking for a transcode at some other bitrate, so you pay in CPU for a stream that still buffers. The arithmetic is unforgiving: a 40 Mbps file needs 40 Mbps of sustained upload, and residential connections are asymmetric by design.
Set the limit per user, not globally: each user's policy has an internet streaming bitrate limit in Mbps, so local viewers keep full quality while remote accounts are capped.
Set the server wide fallback too: Dashboard, Playback holds an internet streaming bitrate limit that applies to anyone you have not configured individually.
Measure your upload before choosing the number: total concurrent remote streams must fit inside your real upload speed, not your download speed, and leave headroom for everything else in the house.
Accept the transcode you just chose: a capped remote stream is a deliberate, predictable transcode, which is exactly the workload hardware encoding handles well.
Teach the client side setting as well: every Jellyfin app has a quality selector with preset steps, and a family member who picks the top entry overrides good intentions unless the server policy caps them.
Watch for the wrong ceiling: set it too low and remote viewers get soft video and a permanent transcode on files that could have direct played.
One capped stream beats one unlimited stream that never plays.
Anti-pattern 4: pointing the Jellyfin transcode cache at the wrong disk
Every transcode writes HLS segments to disk continuously, while several viewers read them back at the same time. Put that directory on the same spinning drive your media lives on and you now have random writes competing with sequential reads for the same heads. The transcode falls behind, the client buffers, and the dashboard still shows the GPU barely working. It looks like a CPU problem. It is a storage problem.
Know where it currently points: Dashboard, Playback, Transcoding has a transcode path field, and the official container expects a /cache mount for it, so an unmapped cache means the segments land inside the container filesystem.
Keep it off the media array: an SSD, even a small one, isolates the write load from the drives serving direct play sessions.
Consider a tmpfs, with a size cap: mounting the transcode directory as tmpfs puts segments in RAM, which is the fastest option, but it consumes real memory and must be given an explicit size limit or a long film can fill it.
Never use a network share: SMB or NFS latency on thousands of small segment writes is the worst case, and a dropped mount kills every session at once.
Do not run it on the boot USB stick or SD card: the write volume of routine family viewing wears cheap flash quickly.
Enable throttling and segment deletion: both live in the same transcoding settings page, and together they stop a paused session from transcoding an entire film ahead of the viewer and filling the disk.
Check free space with df -h on the cache path before blaming anything else. A full cache disk fails playback silently.
Which audio codecs and containers quietly force your family's clients to transcode?
Video gets the attention, but audio and container mismatches are what convert a direct play into something else on phones, browsers and cheap streaming sticks. The good news is that an audio only conversion is comparatively cheap. The bad news is that some clients respond to an unsupported audio track by requesting a full transcode, and lossless audio tracks are enormous, so remote sessions stall on bandwidth even when the video is untouched.
Lossless formats are the usual trigger: Dolby TrueHD and DTS-HD Master Audio pass through to an AV receiver over HDMI, and almost nothing else in the house decodes them.
Browsers refuse Dolby Digital: AC-3 and E-AC-3 are not decoded by desktop Chrome or Firefox, so laptop viewers get an audio conversion to AAC on nearly every film ripped from disc.
MKV is a container problem, not a codec problem: browsers play MP4 natively, so an MKV file is repackaged before it reaches them, which is cheap but only works if the streams inside are supported.
Phones and tablets downmix anyway: a 5.1 or 7.1 track becomes stereo on a two speaker device, and that downmix is a re-encode.
The fix is a second audio track, not a second file: add a compatible stereo AAC track with ffmpeg -i input.mkv -map 0 -c copy -map 0:a:0 -c:a:1 aac -ac 2 -b:a 192k output.mkv, keeping the original for the receiver.
Set the default audio track per user language preference so casual viewers land on the compatible track without choosing it. The living room keeps its lossless soundtrack, and everyone else stops generating work.
