When Eviction Patterns Reveal Hidden Capacity Bottlenecks

You glance at your dashboard. Eviction rate spiked overnight. The pod count dropped by 12% in three hours. Your first thought: memory pressure. But when you check actual memory usage, it's only at 60%. So what's going on?

That's when you start looking closer at the eviction patterns—not just the number of evictions, but the why and how. Because evictions aren't always about running out of RAM. Sometimes they reveal a bottleneck you never knew you had: CPU throttling that delays memory reclaim, I/O congestion that stalls page fault handling, or cgroup limits that are too tight for your workload's burst profile. This article walks through the telltale signs of hidden capacity issues hiding inside eviction logs, with practical steps to diagnose and fix them before they turn into outages.

Why Eviction Patterns Matter More Than Raw Resource Usage

The gap between utilization metrics and actual pressure

Most teams stare at the wrong numbers. They watch CPU at 45%, memory at 62%, disk I/O comfortably under threshold — and declare the system healthy. Then a pod gets evicted. Then another. The dashboard still reads green. That’s the trap: utilization metrics average away the spikes. A server can sit at 70% memory usage for hours, humming along, because the kernel is swapping, the page cache is shrinking, and the pressure is invisible to your Prometheus queries. Meanwhile, the eviction log tells a different story — one of real-time starvation. I have debugged clusters where memory pressure sat at 92% for thirty seconds, triggered evictions, then dropped back to 65% before any alarm fired.

How eviction logs expose resource contention that dashboards hide

Raw resource graphs flatten chaos into smooth lines. Eviction logs capture the seams where the system actually breaks. Think of it this way: your car’s temperature gauge shows normal right up until steam billows from the hood. Eviction patterns are the steam. They reveal moments when a node’s kubelet decided something had to go — not because average usage was high, but because a burst of writes collided with a cron job at exactly the wrong millisecond. The catch is that most capacity planning relies on 95th percentile or max-of-window metrics, which discard exactly these collisions. Evictions don’t lie — they’re the system choosing what to kill.

“We spent two weeks tuning memory limits. The real fix was adding one line of I/O throttling. The eviction log told us first; the memory dashboard never did.”

— Infrastructure engineer, post-mortem for a retail platform that crashed every Black Friday

Real-world example: a 70% memory usage server that started evicting pods

The server showed 70% memory used. Steady for days. Then at 2:14 AM every Wednesday, evictions started — pods dying, alerts firing, on-call paged. Raw memory utilization never crossed 78%. So what changed? The application was using transparent huge pages, and the kernel’s memory reclaim path hit a latency wall when fragmentation grew. The eviction pattern wasn’t about total memory — it was about availability of contiguous pages under allocation pressure. No dashboard at the time measured that. We fixed this by reducing huge page sizes and adding a small swap device (counterintuitive, I know). The evictions stopped. The 70% memory number stayed exactly the same. That hurts — you realize you’ve been optimizing against a phantom metric while the real pressure source hid in plain sight. Raw resource usage is a rearview mirror. Eviction patterns are the engine knocking.

What Eviction Patterns Actually Tell Us (in Plain English)

The difference between hard limit evictions and reclaim-driven evictions

Not all evictions are born equal. I have sat through too many war rooms where an engineer slaps a dashboard on screen and says, "We’re evicting — problem." Wrong order. There is a canyon between a pod that dies because it hit its memory hard limit — the cgroup OOM killer, no negotiations — and a pod that gets evicted because the kubelet tried to reclaim resources from a node under pressure. Hard limit evictions are final. They mean your container asked for 512 Mi, consumed 700 Mi, and the kernel shot it. No second act. Reclaim-driven evictions, however, are the kubelet playing housekeeper: it spots that the node is starved for memory or disk, picks a pod that can be restarted safely, and kicks it out so the node can breathe. That hurts less — but only if you catch it early enough.

The tricky bit is that most monitoring tools color both red. You see a pod eviction event, panic, and add more memory. But if the eviction was reclaim-driven — say, because a bursty cron job filled /tmp and the node hit ephemeral-storage pressure — doubling RAM does nothing. You just wasted a resizing cycle. I once watched a team scale a database cluster from 8 GB to 32 GB per node three times before they noticed the eviction reason field said "NodeHasDiskPressure." Not memory at all. That's the cost of conflating the two types.

Why bursty workloads trigger evictions that average metrics miss

Average CPU sits at 45%. Memory at 60%. The dashboard is green. Yet every seventy-three minutes a pod vanishes. How? Because averages are lies. A workload that spikes to 90% memory for four seconds — then idles — looks tame on a one-minute average, but the kubelet sees the spike in real time. It queries the cgroup every hundred milliseconds or so. That four-second burst is enough to push the node over the eviction threshold, and boom: a pod gets the boot. Most teams skip this: they smooth metrics to reduce alert noise, and in doing so they blind themselves to the actual pressure signal.

What usually breaks first is the garbage collector or a batch job that wakes up, slurps memory, and finishes before the five-minute average window even flinches. The eviction pattern is the only trace. I have fixed two production incidents by looking at the eviction timestamps and matching them to cron schedules — the team had no idea their hourly cleanup script was briefly doubling memory usage. Average utilization said 55%. The truth was "dangerous for 3% of the hour." That's the gap evictions fill.

Flag this for redis: shortcuts cost a day.

Flag this for redis: shortcuts cost a day.

Bursts don't show up in averages. They show up in dead pods and confused on-call engineers.

— observation from debugging a Redis cluster that evicted every 18 minutes for weeks

How to read the eviction reason field in Kubernetes events

Most engineers never open the eviction event JSON. They see a pod disappear and immediately taint the node or resize the deployment. That's like seeing smoke and guessing the fire is in the kitchen — could be, but check first. The eviction-reason field is your flashlight. NodeMemoryPressure means the kubelet thinks the node is drowning in memory usage. NodeDiskPressure means disk space or inodes are running thin. NodePIDPressure means you exhausted process IDs — rare but devastating on churning microservice hosts. And then there is node.kubernetes.io/unreachable, which isn't really an eviction; it's the node controller assuming the machine died and moving pods off a corpse. That's a different class of problem entirely.

The catch is that the reason alone is not enough. You need the eviction-time and eviction-age to see if it's a single spike or a repeating pattern. I have seen teams add 16 GB of RAM because one pod was evicted — only to find out the eviction reason was NodeDiskPressure caused by a log rotation bug that filled the root partition. The pod didn't need memory; it needed a bigger ephemeral volume or a log cleanup script. The eviction reason told us where to look. The pattern told us how often to expect it. Together they reveal the bottleneck without guessing. Next time a pod disappears, read the event. Not the dashboard. The event.

Under the Hood: How Eviction Decisions Are Made

Linux kernel's memory reclaim hierarchy: kswapd vs direct reclaim

Every eviction starts with a memory crunch at the kernel level—long before any orchestrator steps in. The Linux kernel runs two reclaim paths: kswapd, a background daemon that wakes when memory pressure crosses a watermark, and direct reclaim, which slams the brakes on the calling process when kswapd can't keep up. Most teams miss this: kswapd is supposed to be the early warning system. Direct reclaim is the panic button. On a healthy node, kswapd cycles through inactive pages, swaps out cold data, and never lets pressure hit the threshold where processes get blocked. That's the theory. In practice—especially under container orchestration—direct reclaim fires far too often, and each firing stalls that thread for milliseconds. Dozens of threads stalling simultaneously? The node appears healthy in CPU metrics, but latency spikes and evictions cascade.

The catch is that kswapd's behavior changes when memory.high or memory.max cgroup limits are set. The kernel doesn't treat container memory as a single pool; it sees each cgroup as a separate reclaim domain. One container hitting its limit triggers direct reclaim inside that cgroup only. Other containers remain unaware. But the orchestrator—Kubernetes, for instance—monitors the node level. So a single greedy pod can cause local reclaim storms that look like node-wide trouble. I have debugged clusters where kswapd was asleep, direct reclaim was constant, and the only clue was a subtle rise in pgscan_direct metrics nobody was graphing. That's the hidden bottleneck: you chase CPU or memory averages while the kernel is silently burning cycles on page reclaim.

cgroup limits and the OOM killer's role in evictions

Not all evictions are OOM kills—most are quieter. When a container exceeds its memory.limit_in_bytes, the kernel doesn't instantly kill it. It throttles allocation attempts, forces reclaim, and only after repeated failures does it invoke the OOM killer. That gap—between hitting the limit and dying—is where eviction patterns take shape. The orchestrator might preemptively evict the pod before the OOM killer fires, based on node-pressure conditions. But here's the pitfall: the OOM killer's victims aren't always the largest memory consumers. The kernel uses an oom_badness heuristic that prioritizes score, not usage. A tiny sidecar with high oom_score_adj can get killed while the real hog survives. I've seen a logging sidecar evicted every ninety seconds because it sat next to a Java process that never reclaimed memory fast enough. The pattern looked like a scheduling failure. It was a cgroup limit conflict that no dashboard flagged.

“Memory limits don't cause evictions—the gap between the limit and actual working set does. That gap is invisible unless you measure reclaim activity.”

— SRE who spent three weeks on a phantom OOM issue

The orchestrator's eviction signals—memory.available, nodefs.available, imagefs pressure—are all derived from kernel events. If those events are noisy (frequent kswapd wake-ups, shallow reclaim depth), the kubelet may evict pods even though memory usage looks stable. This is the most common misdirection: 75% memory usage on the node, but constant pod evictions. The real pain is reclaim lag, not absolute usage.

How CPU throttling delays memory reclaim and triggers evictions

Memory reclaim is not free—it consumes CPU cycles. Compress pages? That's CPU. Swap pages in/out? CPU. Scan the LRU list? Also CPU. Now add CPU throttling from container limits or cgroup cpu.shares. A pod hitting its CPU quota spends more time waiting for cycles, which means its reclaim work gets delayed. While it waits, memory pressure builds. By the time the kernel can reclaim, the pod has already been marked for eviction by the node's pressure monitors. The relationship is asymmetric: a bursty CPU workload can cause memory evictions even if memory usage never spikes. I fixed one cluster by removing CPU limits on a database pod—the eviction rate dropped to zero overnight. Counterintuitive? Yes. But the kernel's memory manager needs CPU headroom to keep the books clean.

Most teams skip this: CPU throttling metrics are usually averaged over seconds. Evictions happen in milliseconds. So your graphs show the pod was fine—95th percentile CPU usage at 60%—but micro-bursts of throttling delayed reclaim by ten milliseconds, and the orchestrator evicted before the pod could catch up. That hurts. The fix isn't always more CPU; sometimes it's setting cpu.cfs_period_us lower or switching to latency-optimized reclaim modes (vm.zone_reclaim_mode=1). But the point stands: eviction patterns often trace back to the kernel reclaim pipeline, not raw resource exhaustion. Read the reclaim metrics—pgscan_kswapd, pgscan_direct, allocstall—before you resize anything. That's where the real bottleneck lives.

Field note: redis plans crack at handoff.

Field note: redis plans crack at handoff.

A Real Walkthrough: The Database Cluster That Evicted Every 5 Minutes

Initial symptoms: eviction rate spikes with normal memory usage

The team ticketed it as 'mystery evictions' — every five minutes, like clockwork, the Redis cluster shed keys. Memory sat at 58% utilization. No OOM killer ran. The allocator logs showed nothing unusual. Most teams skip this: they see free pages and declare the system healthy. Wrong order. The eviction rate plotted against time revealed a sawtooth pattern: a rapid spike, a 45-second calm, then another spike. I asked the engineer to check who was doing the evicting — kernel or userspace daemon. 'Kernel,' she said. That narrowed it. Userspace eviction patterns usually correlate with memory pressure above the watermark; kernel-driven eviction meant the page reclaim path was failing fast and failing often.

Digging into kernel logs: direct reclaim stalls from CPU throttling

We pulled /proc/zoneinfo and found the low watermark untouched. Pages free, pages available — numbers that should have made eviction dormant. But the kernel was still calling direct reclaim. Why? The trace showed try_to_free_pages() running, then stalling — not because of page scarcity but because the reclaim thread itself was being descheduled. That hurts. The cgroup's cpu quota had been set to a fraction of a core during a 'cost-saving' config push three weeks earlier. Every five minutes, a batch job hit the same cgroup, consumed the entire cpu slice, and the kswapd process got throttled mid-scan. The reclaim path resorted to scanning the same hot pages repeatedly, finding nothing evictable, then giving up — triggering an eviction burst as memory fills the gap between scans. The fix? We raised the cpu shares from 128 to 512 for the database cgroup. Evictions dropped to zero within two hours.

'We spent three days chasing phantom memory leaks. The real villain was a cpu quota smaller than a postage stamp.'

— site reliability lead, after the rollback

Root cause: cgroup cpu shares too low, throttling the reclaim path

The catch is that eviction patterns often point away from memory entirely. Here the pattern was regular, symmetrical, and independent of allocation rate — a dead giveaway that something outside the memory subsystem was blocking the reclaim machinery. Most capacity monitoring tools would show 'cpu idle' because the throttled reclaim threads don't register as busy cycles. That said, one tell remains: if your eviction rate looks like a square wave and your free memory stays flat, stop looking at meminfo. Check cpu usage in the root cgroup, then check cpu.stat for throttled periods. The trade-off is ugly: giving more cpu shares to the database cgroup means stealing cycles from batch jobs. So we pinned the batch workload to a separate cgroup with its own limit. Not a silver bullet — we lost 12% batch throughput — but the database stopped evicting. And that saved the Monday morning rush when traffic doubles. Start tomorrow: graph eviction events alongside nr_throttled for each cgroup. If you see the pattern I described, you have hidden cpu starvation, not a memory bottleneck.

Edge Cases: When Evictions Lie (or Mislead)

Swap-backed evictions that don't mean memory pressure

You stare at the eviction counter. It's climbing every few seconds—clear memory distress, right? Not always. I once debugged a Redis cluster where the eviction rate hit 2,000 events per minute, yet memory utilization sat at a comfortable 65%. The culprit? The kernel had started swapping out anonymous pages to make room for the page cache. The application never truly needed more RAM—it just got caught in a swap storm triggered by a cron job that ran a massive find operation on a virtual filesystem. Swap-backed evictions look identical to genuine capacity pressure in most monitoring dashboards. The eviction pattern screams "buy more memory," but the real fix was tuning vm.swappiness from 60 down to 10 and moving that cron job to off-peak hours. Misread this signal, and you'll throw hardware at a software problem.

That's the dangerous part: eviction events are a measurement of kernel activity, not application need. A process that's been idle for hours can get swapped out, and when another process needs that page, the eviction count ticks up. But your active working set never grew. The metric lies because it conflates two different kinds of pressure: actual demand for more memory versus the kernel's greedy page cache behavior. Most teams skip this: check /proc/meminfo for SwapCached and SwapTotal before trusting those eviction graphs. If swap usage is nonzero while free memory is above 20%, your eviction pattern is a phantom.

NUMA node imbalances causing evictions on one socket only

Here's a scenario I've seen three times in production: a 48-core server evicts pages steadily from NUMA node 0 while node 1 sits at 70% usage. The global eviction rate looks moderate—acceptable, even. But one socket's workload is thrashing. This happens when a multithreaded application allocates memory on the local node of the first thread, then spawns workers that end up pinned to the opposite socket. The kernel tries to satisfy allocations from local memory first; when node 0's memory runs dry, it starts evicting pages to make room, even though node 1 has gigabytes free. The eviction pattern says "memory pressure," but the root cause is a NUMA imbalance, not overall capacity exhaustion.

What usually breaks first is tail latency. One socket's workers suffer page faults while the other socket's workers hum along. The fix isn't more RAM—it's either numactl --interleave=all or restructuring thread-to-core affinities so allocations spread evenly. I've fixed this by adding a two-line numactl wrapper to the application launcher and watching evictions drop to zero. The catch is that most eviction monitoring tools aggregate counters across all NUMA nodes. You need numastat -c or per-node /sys/devices/system/node/node*/vmstat to catch the lie.

Kernel bugs that produce phantom eviction events

Sometimes the eviction pattern is a ghost story. Linux kernel versions 5.4 through 5.10 had a known issue in the LRU list balancing code where reclaim would scan inactive pages even when no memory pressure existed. The result? Consistent eviction events at regular intervals—every 60 seconds like clockwork—with zero performance degradation. The eviction counter ticked up, memory was fine, swap was empty, and NUMA was balanced. But the metric caused a false alarm that triggered a capacity planning meeting. Honest—I sat through a 45-minute call about "memory growth trends" that were just kernel accounting noise. The fix came in 5.10.53 and 5.11.19: a patch that fixed lru_add_drain_all triggering unnecessary reclaim. If your eviction events happen in precise, predictable bursts and correlate with no latency or CPU change, suspect the kernel version, not your workload.

'Eviction counts are like engine warning lights — they tell you something happened, not whether the thing matters.'

— paraphrased from a kernel engineer during a 2023 LSFMM discussion on reclaim heuristics

Flag this for redis: shortcuts cost a day.

Flag this for redis: shortcuts cost a day.

The real pitfall: these phantom events accumulate in your monitoring time series, skewing trend analysis. A 30-day eviction graph that shows a gradual slope might actually be a flat line with a kernel bug superimposed. You make capacity decisions based on noise. The workaround? Cross-reference eviction events with pgscan_kswapd and pgsteal_kswapd counters. If those stay flat while evictions climb, that's a red flag—your evictions are not coming from real reclaim pressure. And if you're on an older kernel, patch it. Not because the bug hurts performance today, but because it blinds you to the moment when real pressure arrives.

The Limits of Relying on Eviction Metrics Alone

Eviction events are reactive, not predictive

An eviction is a scream—not a diagnosis. By the time the kernel or the application layer kicks out a page, a row, or a cache entry, the damage has already started: a query slowed, a connection queued, a user refreshed the page. I have seen teams treat eviction rates like a thermometer, adjusting thresholds up or down, only for the real problem—a memory leak in an adjacent service—to stay hidden. Eviction metrics tell you something was reclaimed, but they never tell you how close to the edge you were before the reclaim happened. That distinction matters. A system evicting 100 objects per second under steady pressure is different from a system that evicts 100 objects per second because a cron job flushed the entire buffer pool. Same number. Different root cause. One is chronic; the other is a spike that self-corrects. Watching evictions alone, you will chase ghosts.

The catch is that eviction counters reset after every event. They have no memory of how long the system held pressure before the release. So you end up with dashboards that look identical for a healthy cluster and one that's hanging on by a thread. False comfort. Or false panic.

False positives from transient reclaim storms vs chronic pressure

A reclaim storm happens when multiple processes suddenly compete for the same resource—think of a cache warming event after a deployment, or a background batch job that scans a full dataset. Eviction rates spike, alarms fire, someone pages the on-call engineer at 3 AM, and by the time they log in, the storm has passed. The system is stable again. The eviction graph shows a jagged peak, but the damage is done: an hour of lost sleep and a growing distrust of your monitoring. I have watched this exact scenario play out at least four times across different clients. The eviction metric is technically correct—it recorded the events—but it misled the operator into thinking the cluster was at capacity when it was only experiencing a transient imbalance.

How do you tell the difference? Pressure Stall Information (PSI). PSI reports how long processes were waiting for memory, IO, or CPU—not just that something was evicted. If evictions are high but PSI stays near zero, the system reclaimed resources without making anything wait. That's a transient event. If PSI shows 10% or more of time stalled on memory allocation, you have chronic pressure. Eviction rates without PSI are like a fire alarm that goes off in a kitchen with burnt toast—loud, but useless when there is an actual electrical fire in the basement. Supplement eviction data with stall averages (10s, 60s, 300s windows) and you stop reacting to noise.

“The scariest eviction pattern I debugged turned out to be a harmless artifact of a logging library changing its retention policy.”

— production engineer, internal post-mortem notes

When to complement eviction data with other signals (like PSI)

Most teams skip this: they graph evictions, set a static threshold, and call it capacity planning. But evictions are a lagging indicator. They describe what already failed, not what is about to fail. PSI gives you a leading view—it measures resource pressure before the system resorts to eviction. Pairing the two lets you distinguish between a system that's healing itself and one that's bleeding out. Add allocation rates and page cache hit ratios, and you build a layered picture. Evictions high, PSI low, allocation flat? Probably a reclaim storm. Evictions moderate, PSI climbing, cache hit ratio dropping? You're running out of headroom. The action changes: first case, ignore and tune alert thresholds; second case, scale memory or reduce working set.

One more pitfall: eviction metrics from different layers don't aggregate cleanly. Kernel page evictions, Redis key evictions, and application-level LRU drops measure different things at different time scales. Rolling them into a single panel makes the dashboard clean but misleading. Keep them separate. Compare them to their respective PSI counters. That sounds like extra work—it's—but it beats waking up at 3 AM for burnt toast.

Reader FAQ: Eviction Patterns and Capacity Bottlenecks

How do I set eviction thresholds that don’t cause false alarms?

Start with your worst five-minute window — not your average. I have seen teams plot a line at 80% memory usage, only to page at 3 AM because a routine batch job brushed against the ceiling. The trick is to layer two signals: a rate threshold (evictions per second) and a duration floor (sustained for 60 seconds). A single spike? Probably a burst. A ten-minute plateau? That’s a leak or a missing index. False alarms drop when you ignore the first blip and chase the pattern. The catch is — you need at least two weeks of historical eviction data to calibrate. Without that baseline, you're guessing.

What does it mean if evictions happen but resource usage is low?

Your monitoring is lying to you. Or rather, it’s telling the truth about the wrong metric. Low CPU and memory can coexist with heavy evictions when the bottleneck lives in the eviction policy itself. Example: a Redis cluster with plenty of RAM but a volatile-lru policy that scans every key before deciding what to kick. The evictions don’t fire because memory is tight — they fire because the scan interval creates a window where new writes pile up faster than the daemon can select victims. We fixed this once by switching to allkeys-lfu and adding a write queue limiter. Eviction rate dropped 70% overnight. Raw resource usage never changed. What usually breaks first is the eviction path, not the hardware.

Can evictions be a healthy sign — say, during burst reclaim?

Yes — but only inside a well-defined guardrail. Short-lived eviction bursts during traffic spikes are the cache doing its job: trading cold data for hot data. That's mechanical sympathy, not a problem. The unhealthy version is when the reclaim rate exceeds the fill rate for more than a few seconds. Honest — a healthy cache evicts 1–2% of its entries per minute under load. Beyond that, you're paying for writes that never get read. One team I consulted called this “thrashing with extra steps.” They had set maxmemory-policy to allkeys-random because they thought randomness would spread wear evenly. Instead, every spike triggered a cascade: evict, miss, fetch from disk, evict again. The fix was a small TTL on stale keys — not more RAM. Adding capacity hides the real issue.

“The hardest eviction pattern to debug is the one that only shows up when the incident is already over.”

— systems engineer, after a post-mortem that blamed the database but was really a connection pool limit

Bottom line: treat evictions as a qualitative signal, not a binary health check. Pair them with latency percentiles and write-ahead-log depth. If evictions rise but p99 latency holds steady, you're probably fine. If latency jumps by 40 ms — even once — the eviction path is your limiting factor. That's the hidden capacity bottleneck.