Transparent Huge Pages + Postgres: Stop P99 Latency Cliffs [2026]

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Transparent Huge Pages (THP) Postgres performance problems almost never show up as “the database is slow.” They show up as a system that looks fine on averages and then occasionally falls off a cliff at p99. Suddenly you’ve got “random” timeouts, queue backups, and a pager that hates you.

I’m going to be blunt. If you run self-managed Postgres on Linux and you haven’t made an explicit decision about THP, you’re accepting tail latency risk for no good reason. The kernel is doing work you didn’t ask for, at the exact moment you can least afford it.

This post is the runbook I wish more teams had: detect → measure → change → validate → enforce. Especially if you’re running Postgres on Kubernetes nodes where config drift is basically a law of nature.

Key takeaways

• THP is not the same thing as explicit Huge Pages (hugetlbfs). Confusing them is how teams “tune huge pages” and still keep the latency cliff.
• For OLTP Postgres, the safest default is THP=never plus defrag=never. That combo stops background compaction from stalling your backends.
• If you choose madvise, you still need to control defrag and verify whether Postgres is actually using THP. Otherwise you’ve just moved the cliff somewhere else.
• Validate with p99/p99.9 latency and pg_stat_statements, not averages. Averages will lie to you.
• In Kubernetes, this is a node guardrail. If you don’t enforce it, it will drift. And then you’ll “mysteriously” rediscover this post during an incident.

If your Postgres p99 is spiking “randomly,” assume the kernel is doing work you didn’t ask for.

What Transparent Huge Pages (THP) is, and why Postgres teams keep getting burned

Transparent Huge Pages (THP) is a Linux kernel feature that automatically backs virtual memory with huge pages (commonly 2 MB) instead of base pages (commonly 4 KB). It can also promote and demote page sizes over time.

The kernel docs are pretty explicit about scope: THP mainly applies to anonymous memory mappings and tmpfs/shmem, and the intent is “performance without app changes” by handling page promotion/demotion automatically. (Linux kernel docs)

On paper, that’s a nice deal. Fewer TLB misses. Fewer page faults. Free speed.

In production with Postgres, it’s usually not a nice deal.

Postgres is latency-sensitive and memory-active under concurrent load. The thing that bites you isn’t the existence of huge pages. It’s the kernel doing automatic promotion plus defrag/compaction work at runtime, on your schedule, not its own.

The kernel docs call out the tradeoff clearly: huge pages reduce TLB misses and reduce page-fault frequency. One page fault per 2 MB region is 512x fewer faults than 4 KB pages. But page faults can become more expensive, and the kernel may need background work to create and maintain huge pages. (Linux kernel docs)

When that background work lands at the wrong moment, you get the classic “latency cliff.” Your median stays boring. Your p99 gets obliterated.

THP vs explicit Huge Pages (hugetlbfs) in Postgres

This is where a lot of “performance tuning” content goes off the rails.

Postgres docs talk about huge pages in the context of kernel resources. In the PostgreSQL 18 docs, 18.4. Managing Kernel Resources is the umbrella section, and 18.4.5. Linux Huge Pages is the part that’s actually about explicit huge pages, typically pre-allocated and managed intentionally. (PostgreSQL docs: 18.4 Managing Kernel Resources)

Explicit huge pages (hugetlbfs) are predictable. You provision them. You monitor them. You know what you’re getting.

Transparent huge pages are the opposite. They’re opportunistic and adaptive. That’s fine for some workloads. For OLTP Postgres, it’s where tail latency goes to die.

Why THP causes Postgres p99 spikes even when the average looks fine

Here’s the failure mode that keeps repeating in real systems:

1. Everything looks stable. Memory pressure is moderate but not catastrophic.
2. THP is enabled in always, so the kernel tries to allocate and maintain huge pages.
3. The kernel background thread (khugepaged) scans memory and tries to collapse 4 KB pages into 2 MB huge pages.
4. When it can’t find a contiguous 2 MB region easily, it triggers compaction/defrag work.
5. Compaction can stall processes. Those stalls show up as tail latency spikes in Postgres queries.

This is not some spooky “kernel magic.” It’s exactly what the THP design implies: promotion/demotion costs real CPU and can involve memory compaction. The sysfs knobs under /sys/kernel/mm/transparent_hugepage/ exist because the kernel authors know these tradeoffs matter. (Linux kernel docs: THP sysfs knobs and modes)

So why do averages look fine?

• Most queries never hit the stall.
• When it happens, it’s intermittent.
• Your dashboards default to rolled-up means because observability vendors love pretty lines.

And why does p99 explode?

• A few unlucky backends get paused during compaction.
• Under load, one stalled backend turns into queues, lock waits, and cascading slowdowns.

Tail latency is where “the DB is fine” turns into a user-facing outage.

Exact THP modes and the recommended settings for Postgres OLTP

Linux exposes two knobs you actually care about for this topic:

• /sys/kernel/mm/transparent_hugepage/enabled
• /sys/kernel/mm/transparent_hugepage/defrag

The modes you’ll see are:

• always: the kernel aggressively uses THP
• madvise: the kernel uses THP only when applications request it via madvise()
• never: the kernel doesn’t use THP

Those names aren’t blog-invented. They’re straight from the kernel docs. (Linux kernel docs)

My opinionated defaults (the ones that stop latency cliffs)

If you’re running OLTP Postgres (user-facing requests, spiky concurrency, real SLOs), the default that prevents most THP-induced cliffs is:

• enabled = never
• defrag = never

If you have a specific reason to keep THP around (some analytics-heavy mixed workloads, some VM setups), the least dangerous option is:

• enabled = madvise
• defrag = never

If you’re currently on always, you’re basically telling the kernel: “Feel free to do background memory work whenever you want.” That’s not a production posture. That’s a dare.

Safe defaults matrix (what you can actually change)

Step 1: Check system-wide THP usage (and don’t guess)

You want two facts, not vibes:

1. Current mode (enabled)
2. Current defrag behavior (defrag)

On most distros:

• cat /sys/kernel/mm/transparent_hugepage/enabled
• cat /sys/kernel/mm/transparent_hugepage/defrag

You’ll see something like always madvise [never] where the brackets mark the active mode.

That’s the kernel’s sysfs interface doing exactly what it says on the tin. (Linux kernel docs: THP sysfs knobs and modes)

Also check whether khugepaged is active and burning CPU during incidents:

• ps -eo pid,comm,pcpu,pmem,args | grep khugepaged
• top / htop during a spike and look for khugepaged

If khugepaged CPU jumps at the same time your p99 jumps, that’s not “proof,” but it’s a pretty loud hint.

Step 2: Check THP usage per process (is Postgres actually using it?)

System-wide mode is necessary, but it’s not sufficient. You also want to know what’s happening inside the Postgres process.

At minimum:

• Identify the Postgres PID (or the postmaster PID)
• Inspect /proc/<pid>/smaps or /proc/<pid>/smaps_rollup and look for huge page indicators

On RHEL-family systems, Red Hat’s runbook includes practical steps for checking system-wide and per-process THP usage. This is one of those rare cases where the boring enterprise doc is the right reference because it’s written for operators, not for Twitter. (Red Hat KB 46111)

Why you care:

• If you set madvise but nothing in your stack requests THP, you might be chasing a ghost.
• If you set never but a tuned profile flips it back on, you’ll want receipts.

Step 3: Benchmark before you change anything (pgbench + p99)

If you change kernel memory behavior in prod without a baseline, you’re doing performance cosplay.

Here’s the loop I like:

1. Capture p50/p95/p99/p99.9 latencies from your app or from pgbench.
2. Capture a 5–15 minute window of pg_stat_statements.
3. Make the THP change.
4. Repeat the same measurement window.

You’re looking for a very specific shape:

• Median might not move much.
• p99/p99.9 should smooth out.
• Query max times in pg_stat_statements should drop.

Use an actual number for the run. For example: run pgbench for 10 minutes at a concurrency of 64 and capture latency percentiles.

If your app has a known “spiky hour” (batch jobs, traffic peak, cron chaos), measure during that. Don’t benchmark at 2 p.m. and declare victory.

Step 4: Apply the fix — disable THP at run time (immediate mitigation)

When you’re in an incident, you want the change now, even if it won’t persist.

Runtime mitigation:

• Write never to /sys/kernel/mm/transparent_hugepage/enabled
• Write never to /sys/kernel/mm/transparent_hugepage/defrag

This is the operational equivalent of pulling the emergency brake. It won’t survive a reboot. That’s fine. The goal is to stop the cliff and stabilize.

Red Hat’s procedure covers the runtime disable path and it’s worth following if you’re on RHEL. (Red Hat KB 46111)

Step 5: Make it persistent — disable THP at boot time (the real fix)

If you don’t persist it, it will come back. And yes, it will come back at 3 a.m.

You’ve got a few durable options. Which one you use depends on what your fleet standards look like.

Option A: GRUB kernel parameters (works broadly)

Set the kernel command line so THP is disabled at boot.

Common pattern:

• Add transparent_hugepage=never to the kernel cmdline

Then regenerate GRUB config and reboot.

This is the most reliable “set it and forget it” method across distros.

Option B: systemd-tmpfiles (cleaner than rc.local)

On systemd hosts, you can write the sysfs knobs at boot via a tmpfiles rule.

This avoids resurrecting rc.local hacks and keeps the change declarative.

Option C: tuned profiles (RHEL ecosystems)

If you use tuned, be careful. Some profiles override THP behavior. The worst case is thinking you disabled THP and then a tuned profile quietly flips it back.

Again, Red Hat is explicit about monitoring and disabling THP in an enterprise environment. Follow it if that’s your world. (Red Hat KB 46111)

Step 6: Validate with pg_stat_statements (Postgres 18/19-era workflow)

If you’re not using pg_stat_statements, you’re debugging Postgres blind.

The official docs are clear: pg_stat_statements must be loaded via shared_preload_libraries (so yes, you need a restart). It tracks planning and execution stats, and it relies on query identifier calculation, which is enabled when compute_query_id is auto or on. (PostgreSQL docs: pg_stat_statements)

That’s the 2026 angle that actually matters here: newer Postgres makes statement-level instrumentation less annoying, so you have fewer excuses.

F.32. pg_stat_statements — track statistics of SQL planning and execution

That’s literally the section title in the docs, and it’s exactly what you want. You’re not chasing a slightly faster mean. You’re trying to stop unpredictable stalls.

F.32.1. The pg_stat_statements View

The view has the columns you need to validate tail improvements: min/mean/max times, total time, and call counts. Rows are keyed by (dbid, userid, queryid, toplevel). (PostgreSQL docs: pg_stat_statements)

What I pull in practice:

• Top queries by total execution time (to see if anything regressed)
• Top queries by max execution time (to catch tail cliffs)
• A before/after comparison window around the change

Also: reset stats when you do controlled experiments. Otherwise you’re mixing pre-change and post-change behavior and pretending it’s science.

Step 7: Validate with p99/p99.9 latency, not just statement stats

pg_stat_statements is necessary, but it’s not the same thing as end-to-end latency.

What I like to validate after the change:

• App-level request p99 and p99.9 over at least 24 hours
• DB pool wait time (if you have it)
• Postgres wait events distribution (newer Postgres versions make this more useful)
• OS signals: khugepaged CPU drops, fewer compaction stalls

If p99 got better but throughput dropped, you may have traded one bottleneck for another. That’s not the common outcome with THP changes, but don’t assume you’re immune.

Step 8: Kubernetes guardrails — enforce THP settings on nodes and prevent drift

In Kubernetes, the most common failure mode is trying to “disable THP inside the container.” That’s not how kernels work.

THP is a node-level setting. Treat it like vm.swappiness or overcommit policy. It lives with the node.

Two patterns that actually work:

Pattern 1: Node bootstrap / machine config (preferred)

• If you run OpenShift, use a MachineConfig to set kernel args.
• If you run Cluster API or managed node groups with custom AMIs, bake the kernel args into the image.

This is the highest-integrity approach. Nodes come up correct.

Pattern 2: Privileged DaemonSet (enforcement + audit)

If you can’t bake images quickly, use a privileged DaemonSet that:

• Checks /sys/kernel/mm/transparent_hugepage/enabled and /defrag
• Writes the desired value if it’s wrong
• Exposes a node condition or metric so you can alert on drift

Be realistic: it’s still a band-aid compared to doing it at boot. But it’s miles better than hope.

Drift prevention checklist

• Alert if any node reports always.
• Alert if defrag is not never.
• Gate node pools. Don’t schedule Postgres pods on nodes that fail the check.

If you’re doing this for a mission-critical database, treat it like a compliance control, not a one-time tweak.

Extra guardrails: kernel knobs that correlate with THP-related stalls

THP issues usually show up next to a couple other kernel behaviors. You don’t need to “tune the whole OS,” but you should know what tends to correlate.

The defrag knob is the big one

If I had to pick a single “stop the cliff” setting besides disabling THP entirely, it’s defrag=never.

That tells the kernel: don’t do expensive compaction work in the background just to satisfy huge page allocations.

Watch for memory pressure and compaction signals

Correlate Postgres p99 spikes with OS-level signals:

• khugepaged CPU usage
• Memory compaction activity
• Swap activity (even small amounts)

If you see compaction and swap during p99 spikes, you don’t have a Postgres problem. You have a node problem.

Putting it together: the copy/paste runbook

Here’s the exact loop I’d run, in order.

1. Record baseline: p99/p99.9 latency for 30 minutes during representative load.
2. Check THP system-wide: read /sys/kernel/mm/transparent_hugepage/enabled and /defrag.
3. Check per-process: confirm whether the Postgres PID is using THP via /proc/<pid>/smaps(_rollup).
4. Enable measurement: ensure pg_stat_statements is on (restart required). (PostgreSQL docs)
5. Mitigate: set enabled=never and defrag=never at runtime to stop an incident.
6. Persist: set boot-time config via GRUB or a node image.
7. Validate: compare before/after p99 and pg_stat_statements max times.
8. Enforce (Kubernetes): machine config or privileged DaemonSet plus alerts.

Conclusion: make THP a deliberate decision, or the kernel will decide for you

The industry has a weird habit of treating kernel settings like folklore. THP isn’t folklore. It has documented modes. It has documented knobs. And for OLTP Postgres, the default Linux behavior is often the wrong one.

So don’t argue about THP in theory. Run the loop. Measure p99. Change one knob. Measure again.

My bet for the next year: as more teams run Postgres on multi-tenant Kubernetes node pools, THP misconfiguration becomes the new “noisy neighbor” incident class. The teams that win won’t be the ones with the fanciest database. They’ll be the ones who turned kernel behavior into an enforced guardrail.

Originally published on kunalganglani.com