Eviction Cycles Before the Outage: Reading the Warning

Evictions in production rarely announce themselves with a bang. They creep in as a steady beat: a pod dies here, a VM migrates there, a container gets OOM-killed just after peak. Most dashboards paint these as isolated blips. But if you zoom out, a pattern emerges—a cycle that repeats, tightens, and often precedes something worse.

This article treats eviction cycles as a diagnostic signal, not a nuisance. We'll walk through who needs to care, what to have in place before you start, a step-by-step workflow, and the tools that make it observable. You'll also get variations for different environments and a heap of pitfalls to avoid. No hand-waving, just practical advice from the trenches.

Who Should Watch Eviction Patterns and What Happens If You Don't

The SRE who ignores evictions until pager duty

You're asleep at 3:47 AM when the first alert fires. Cache hit rate dropped forty percent in the last two minutes. The on-call playbook says restart the service, so you do. It comes back, the metrics stabilize, and you go back to bed. The next night, same thing. By Thursday, you're restarting every four hours and the incident doc has become a novel about memory fragmentation. That hurts. The eviction counter on your dashboard has been climbing for three days, but nobody set an alert on it because evictions felt like background noise.

The truth is, your cache is not failing. It's telling you exactly how it dies. Eviction cycles have a rhythm—slow creep during normal load, then a sharp knee when something changes upstream. I have sat through enough postmortems where the graph shows the eviction rate spiking twelve hours before the outage. Twelve hours of warning, and the pager only went off when users started seeing errors. The SRE who tracks evictions gets a head start. The one who ignores them gets a week of interrupted sleep and a glare from the engineering manager.

The catch is that evictions are not always bad. A least-recently-used cache evicting old entries is healthy. The pattern you need to watch is the slope. Steady state, then a hockey stick. That's the difference between a cache doing its job and one that has become a liability. Wrong order—you watch the hit ratio, not the eviction slope. That's a classic mistake.

Platform teams scaling without eviction visibility

Platform teams have a different failure mode. They scale instances based on CPU and memory utilization, which look fine because the cache is doing what it was told. The eviction rate, however, is a leading indicator. Memory utilization lags. By the time your autoscaler sees pressure, the cache has already discarded the useful entries, and your database is absorbing traffic it should never have seen.

We fixed this once by adding eviction rate to the scaling signal alongside CPU. The difference was stark. One service kept evicting in waves—every time the cache filled, it dumped a batch of entries, the database spiked, and the cluster wobbled. Scaling on evictions smoothed it out. That said, you can't naively scale on evictions alone. A sudden spike might be a legitimate cache flush from a deployment, not a capacity problem. You need the history to tell the difference.

What usually breaks first is the slow bleed. A service grows, traffic patterns shift, and the cache size quietly becomes too small. Nobody notices because the eviction rate is not on any dashboard. The first sign is a database that starts throwing connection timeouts during a routine marketing email blast. That's the cost of reactive firefighting—you spend two hours diagnosing what a five-minute look at eviction trends would have explained.

'It died like it lived—one eviction at a time, until there was nothing left to serve.'

— a senior SRE, three weeks after the incident

The cost of reactive firefighting vs. proactive signal reading

Reactive firefighting feels heroic. It's also expensive. A single outage costs you the on-call engineer's night, the follow-up review meeting, the remediation tickets, and the trust of whoever depends on the service. Proactive signal reading costs you a dashboard and a morning to set it up. I have seen teams resist this because eviction metrics feel abstract. They're not abstract.

The practical difference is measurable. One team I know watches eviction cycles across their fleet and gets a daily digest. They catch capacity issues on Tuesday morning, plan a cache resize, and never mention it again. Another team—same size, same traffic—spent a full sprint rebuilding their cache layer after a preventable outage. Both had the metric available. Only one looked at it.

That's the whole argument. Eviction patterns are a canary. You don't have to be perfect at reading them. You just have to look before the pager forces you to. The warning signs are there. The only question is whether your team is the one that sees them early or the one that explains why the cache was misconfigured for nine months.

What You Need Before You Start Correlating Evictions

Metrics You Must Already Collect: CPU, Memory, Disk, Network

Evictions don't happen in a vacuum. Before you can correlate a single eviction to an outage, you need a full picture of what the machine was doing at that exact moment. CPU saturation, memory pressure, disk I/O waits, network packet drops — all of these leave fingerprints on the same timeline. If you're only tracking memory, you're blind. I have seen teams chase a "memory leak" for two days when the real culprit was a misconfigured network interface causing the app to retry endlessly and churn through connections. Wrong order. Collect all four metrics at a minimum, with a granularity of at least one data point per second for memory and CPU, and five seconds for disk and network. Coarse five-minute averages will smooth away the very spikes that trigger eviction storms.

The trade-off here is storage cost versus signal fidelity. High-resolution metrics eat disk space fast — a single host pushing 20 metrics per second at full fidelity runs you roughly 1.7 GB per day. Most teams balk and drop to 30-second intervals. That's a mistake. Evictions and their precursor conditions often unfold in sub-second windows; a 30-second gap can hide the entire sequence. Find a middle ground: store raw data for 24 hours, then downsample to 10-second averages for 30 days. You'll keep the forensic detail for immediate postmortems and still have trend data for baseline work.

Log Aggregation and Tracing Basics

Metrics tell you what happened. Logs tell you why. You need both, correlated on a shared timestamp. If your eviction logs live in one system and your application logs in another, and neither has a consistent clock, you'll spend hours manually aligning timestamps that drift by milliseconds. Painful. Set up a centralized log aggregator — ELK, Loki, whatever fits your stack — and ensure every eviction event writes out the process ID, the cgroup identifier, and the exact memory pages being reclaimed. Without those details, you'll never distinguish a deliberate cache eviction from a panic-driven OOM kill.

Distributed tracing is the third leg of this stool, and it's the one most teams skip. When an eviction degrades a service on node A, the effects ripple to dependent services on nodes B and C. A trace that spans those hops shows you the propagation path — where latency spiked, which calls timed out, which retries piled up. We fixed a recurring production incident last quarter by correlating eviction timestamps with trace spans that showed a downstream database connection pool exhausting exactly 400 milliseconds after each eviction wave. Without the trace, the connection pool would've looked like the root cause. It was a symptom.

You can't read eviction patterns without a time-synchronized record of system behavior, application behavior, and their interaction points.

— Production engineer, post-incident review

A Baseline of Normal Eviction Behavior for Your Stack

Here's the catch: an eviction spike is only meaningful if you know what "normal" looks like for your workload. A Java service with a 2 GB heap may evict 50 entries per minute during steady state and 5,000 during a deal flow surge — both healthy. The same numbers on a Python service with a small LRU cache would signal a serious problem. So before you can call anything anomalous, you need a baseline. Collect eviction counts, rates, and durations across at least two full weeks of production operation, covering both weekdays and weekends, plus at least one known deployment cycle. That last bit matters — deploys often cause transient eviction spikes as new processes warm up caches, and you don't want to mistake that for a real fault.

The baseline isn't a single number; it's a distribution. Track the 5th, 50th, and 95th percentiles of eviction counts per minute, plus the standard deviation. The 95th percentile is your early warning threshold. When evictions exceed that for five consecutive minutes — not just a single spike — you have a signal worth investigating. That said, the baseline must be re-evaluated after any significant change to your workload: a new feature that doubles memory usage, a traffic shift to a different region, a regression in a hot path. A stale baseline will produce false alarms or, worse, blind you to genuine degradation. Recompute yours every two weeks or after any deploy that touches memory management.

One more prerequisite, often forgotten: your eviction counters must be monotonic and non-resetting across process restarts. If your instrumentation resets counters to zero on restart, the baseline gets corrupted and every deploy looks like a massive eviction drop. We lost a week to exactly that — a counter that reset on graceful restart, producing beautiful downward spikes that meant nothing. Fix that first. Secure a consistent counter, then build the baseline, then correlate. The order matters.

A Practical Workflow for Turning Evictions into a Signal

Step 1: Collect eviction events with timestamps and reasons

Your eviction log is useless if it only says “cache evicted something.” The first thing I do in any production environment is check whether the eviction reason actually gets recorded. Redis gives you evicted_keys as a counter, but that’s a corpse—it tells you something died, not why. You need the reason codes: allkeys-lru versus volatile-ttl versus noeviction hitting its ceiling. Each one points at a different failure mode. LRU evictions mean your working set outgrew memory. TTL evictions mean keys are expiring faster than they’re being consumed. Noeviction means writes are failing entirely—that’s the loudest alarm you’ll ever hear.

Log every eviction with a millisecond timestamp and the key name, if your setup allows it. The key name matters more than people think. One team I consulted had a cache that was evicting thousands of session tokens every minute. The counter looked scary, but the key names showed the pattern was entirely predictable—short-lived objects with a 30-second TTL that were never read again. Harmless. The real problem was a separate class of keys, user profiles held for an hour, that were getting evicted during peak hours. Wrong order. You can't see that separation without key-level granularity.

The catch is that key-level logging costs you performance. Every eviction event written to disk adds latency to your eviction path. In practice, I sample aggressively instead of logging everything. Log the first eviction of each key, then suppress repeats for 60 seconds. You get the shape of the problem without eating your throughput. That trade-off is acceptable—your goal is signal, not archaeology.

Step 2: Aggregate and visualize cycles over time

Raw eviction counts are noise. The signal lives in the cycle. Plot evictions per second on a rolling five-minute window, then overlay that against your request rate and your memory utilization. What you’re hunting for is the lag. Healthy systems show evictions creeping up as memory fills, with a stable relationship to traffic. Unhealthy systems show evictions spiking in waves—each spike creating a thundering herd that doubles the next wave’s height. That’s the death spiral, and it starts hours before the outage.

Most teams skip this aggregation step entirely. They watch the raw counter, see it bump from 2,000 to 2,500 per minute, and shrug. The bump isn’t the warning. The warning is that the time between eviction spikes keeps shrinking. Measure the inter-spike interval. If that interval drops by 30% across a 24-hour period, you’re trending toward a cliff, not a gentle slope. One engineer I worked with caught a production meltdown eight hours early by noticing that eviction spikes—previously every 45 minutes—had compressed to every 12 minutes. The cause was a code deploy that doubled cache write volume. Nobody saw it until the metrics showed the compression.

A simple line chart works. You don’t need Grafana dashboards with seventeen panels. Just put evictions, memory, and request latency on the same timeline. The moment evictions start leading latency—appearing 60 to 90 seconds before response times degrade—you have a leading indicator.

Step 3: Correlate with system metrics to spot leading indicators

Evictions alone are a lagging indicator. Correlating them with other metrics turns them into a prediction. The strongest correlation I’ve seen is between eviction rate and cache miss rate. When evictions go up and misses follow within two minutes, you’re about to see a database load spike. That’s your call-to-action moment. The database hasn’t failed yet, but its query rate is climbing. I’ve watched a PostgreSQL instance go from 40% CPU to 95% CPU in eleven minutes after an eviction burst—and the queue depth was the second thing to blow.

Build your correlation around the ratio: evictions per second / cache hits per second. Under normal load, that ratio sits below 0.01. When it climbs past 0.05, your cache has stopped protecting the backend. That’s not a threshold I invented; it’s a practical number that has caught real incidents. Your mileage will vary, but the principle holds: measure the ratio, trend it, and alarm on the trend’s slope rather than its absolute value.

An eviction spike is a symptom. The disease is always upstream—a code change, a traffic surge, or a data shape shift.

— Runbook note from a senior SRE who’s seen this fail four times

The final step is writing a short-lived alarm. I don’t mean paging someone at 3 a.m. for a five-second spike. Set a window—say, eviction ratio above 0.05 for ten consecutive minutes—and push that to a chat channel, not a pager. If the ratio keeps rising for another ten minutes, then escalate. That two-tier approach keeps your team alert without exhausting them on noise. The goal is to catch the trend early enough to act, not to automate the act itself. What you do with the signal—scaling memory, fixing the code, adding replicas—is a separate decision that still requires human judgment. The workflow just gets you there before the outage, not after it.

Tools and Setup Realities: What Actually Works in Production

Kubernetes: Where the Signals Hide in Plain Sight

Start with the kubelet eviction manager—it’s the first thing that screams before a node goes dark. The kubelet tracks memory, disk, and inodes against hard and soft thresholds, and when it crosses the soft ones it starts evicting pods in priority order. Most teams only discover this after the fact, staring at a terminated pod with a terse Evicted status. That’s the late signal. The early one lives in kubelet_evictions and container_memory_working_set_bytes metrics, which the metrics-server scrapes but rarely anyone visualizes. Prometheus can catch the slope—if you query kubelet_evictions_total over a five-minute window, you’ll see the count climbing long before the node flips to NotReady. Pair that with node_memory_MemAvailable_bytes and you have a leading indicator that beats any dashboard alert.

Field note: redis plans crack at handoff.

Field note: redis plans crack at handoff.

The catch is cluster size. A three-node test cluster gives you clean data. A forty-node production pool drowns you in noise, especially when the metrics-server defaults to a 15-second scrape interval and your retention window is a week. I have seen teams miss the pattern because they graphed the wrong metric—watching node_memory_usage instead of eviction counts. The usage curve barely moves; evictions spike like a heartbeat right before the outage. Set up a recording rule that flags any node with two or more evictions in a ten-minute window. That rule, deployable via a simple PrometheusRule CRD, turns a scattered event stream into a single actionable alert. What usually breaks first is the alert routing—nobody owns the pager, so the signal dies in a Slack channel.

Cloud VMs and Bare Metal: Different Surfaces, Same Panic

On cloud VMs, the guest-agent is your unsung hero. The Azure Linux Agent and the AWS SSM agent both log memory pressure events to /var/log/waagent.log or cloud-init-output.log, but nobody parses them because they’re verbose and timestamped oddly. Yet those logs capture the precise moment the hypervisor began throttling or preparing for migration. The cloud provider’s migration stats—available via the metadata API or the console—show how often the VM was live-migrated. Every migration spikes memory overhead, and if the host is already tight, evictions cascade. I have watched a single migration trigger a domino of OOM kills in a Java service, all because the migration inflated the working set by 200 MB.

Bare metal is rawer. The kernel OOM logs in dmesg are blunt but reliable, and systemd’s cgroup accounting gives you per-service memory limits that the OOM killer actually respects. Run systemd-cgtop under load—it shows you which unit is closest to its MemoryMax, and that’s your eviction candidate. The trade-off: dmesg wraps quickly and logs get lost on reboot. Persist them with pstore or journald’s persistent storage, or you’ll reconstruct the aftermath from memory. That hurts. One production incident taught me to ship kernel logs to a central collector before the crash, not after.

Every eviction signal is a story told twice: once in the logs, once in the downtime. Most teams read only the second version.

— SRE lead, after a 4 a.m. pager storm on a memory-bound worker pool

The setup reality is that none of this works without a scrape cadence tuned to your churn. Five-second scrapes on bare metal waste CPU; sixty-second scrapes on Kubernetes miss burst evictions. Middle ground: fifteen seconds for eviction-related metrics, and let Prometheus’s irate() function smooth the spikes. Most teams skip this tuning and get either false alarms or silence. Wrong order—configure the cadence before you wire the alerts, not after the first incident. Start by exporting one metric to a test dashboard, then expand. What actually works in production is boring: a single node exporter, a truthful recording rule, and a pager that goes off once per shift—not a screen full of red.

Variations for Memory-Constrained, Multi-Tenant, and Other Tight Spots

When you can’t afford full tracing: lightweight logs

Full distributed tracing is a luxury. In a tight memory environment, the agent itself becomes the eviction source—ironic, and expensive. I have seen a 512 MB container die because the monitoring sidecar asked for 200 MB of heap. The fix was embarrassingly simple: strip the trace IDs, keep the counters.

What actually works is a single log line per eviction attempt, emitted at WARN level, with three fields: timestamp, tenant ID, and the number of live objects at that moment. No stack traces. No object class names. You lose the ability to see which allocation failed, but you keep the ability to see when the pattern accelerates. That's the signal that matters. The catch is retention—these lines multiply fast. Rotate them every 15 minutes, not daily. Aggregation is your enemy when memory is scarce.

Some teams go further and write only deltas. If the eviction count between two sampling windows jumped by more than a factor of three, log it. Otherwise, stay silent. That cuts noise by ninety percent. The trade-off is you miss slow, linear creep—the kind that doubles over six hours. Decide which failure mode scares you more. Then build accordingly.

Multi-tenant environments: isolating eviction sources

One noisy neighbor can look like a system-wide death spiral. I debugged a production outage where the eviction graph spiked across all nodes, and everyone assumed a memory leak in the shared cache. Wrong. It was a single tenant running a pathological query, hammering the same cache keys until the entire region thrashed. The pattern was real, but the attribution was backwards.

You need per-tenant eviction counters before you need anything else. Even approximate ones. If you can tag the current thread’s tenant context at the point of eviction—often just a ThreadLocal read—you can separate the herd. The math gets fuzzy when tenants share object pools, but you're not aiming for accounting-grade accuracy. You're aiming for a direction: “this tenant, not that one.”

In shared memory, the loudest evictor is rarely the one that will crash. It's the one whose pattern you can't see.

— Field note, multi-tenant cache post-mortem, 2023

The pitfall here is over-isolation. If you hard-partition memory per tenant, you lose the statistical multiplexing that made shared caching worthwhile in the first place. Soft quotas, not hard limits. Let a tenant borrow from a shared pool, but record the debt. When the debt exceeds twice the quota, you have your signal. That hurts fairness, but it beats a full outage.

Spot instances and preemptible VMs: expected vs. dangerous evictions

Spot instances make this whole exercise weird. Your nodes die on purpose, so a certain eviction rate is normal—even healthy. The danger is conflating reclamation with degradation. A VM that gets preempted at the hardware level will show a sudden drop in memory pressure, not a spike. That's your tell.

Track the direction of the trend, not the absolute number. If evictions rise steadily for 20 minutes before the node disappears, that's a degradation pattern—something inside is leaking or growing. If evictions stay flat and then the node vanishes with no warning, that's preemption. The first is fixable. The second is cost of doing business; you just need your failover to be faster.

One practical adjustment: on spot fleets, lower your alert threshold. A 10% eviction-rate increase means nothing on an on-demand box, but on a preemptible one it's the difference between a clean drain and a hard kill. Set a second, stricter threshold for spot nodes only. Most teams skip this and get paged at 3 AM for behavior they designed into the system. Then they tune the alert off entirely—which is worse. Wrong order. Calibrate first, then automate.

Debugging False Positives and Missed Signals When the Pattern Breaks

Misleading metrics: CPU throttling vs. memory pressure

First rule of eviction forensics: evictions don't tell you *why* memory vanished, only that something clawed it back. A cloud instance that gets CPU-throttled mid-request often *looks* like a memory crisis — latency spikes, connection pools drain, the kernel starts reclaiming pages — but the root cause sits in a completely different subsystem. We fixed a false alarm like this last quarter: graphs showed a steep eviction climb, the whole team braced for an OOM blowout, and the actual trigger was a noisy neighbor hogging the physical host’s CPU. Evictions were a symptom, not a cause.

So before you treat eviction rates as gospel, ask: did *memory pressure* drive the reclaim, or did a slowdown make the app allocate more (retry storms, buffer bloat, queue backups) which then spilled into reclaim? The distinction matters because the fix differs. Throttling gets solved with capacity planning or request prioritization; memory pressure gets solved with heap tuning or cache sizing. Misread the signal and you’ll “fix” the wrong layer.

Alert fatigue: tuning thresholds without losing signal

The second trap is the opposite: you tune your alert so tight that it fires constantly, then you loosen it until it never fires. I have seen this cycle destroy more on-call trust than any actual outage. Start with a static threshold — say, eviction rate above 50 pages per second for five minutes — and you’ll get paged during routine cache churn. Raise it to 500 and you’ll sleep through the slow, creeping eviction pattern that precedes a multi-tenant outage.

The workaround is not a better threshold; it’s a *shape* detector. Look for a rate of change rather than an absolute level. Eviction cycles that climb steadily over 15–30 minutes, then plateau, then accelerate again — that’s the warning. A flat spike that returns to baseline in 90 seconds is usually a batch job or a cold start. Tune for the slope, not the height. Then set a second, higher alert for absolute spikes so you don’t lose the catastrophic case entirely.

When eviction cycles don’t precede an outage: alternative causes

Sometimes the pattern breaks *because* you fixed something. We rolled out a more aggressive page-cache eviction policy, and suddenly every dashboard screamed red — but the app was faster, latency dropped, no errors. The evictions were working as designed; we just weren’t used to seeing them. That’s the false positive nobody prepares for: a healthy system can evict aggressively, and the signal only becomes meaningful when combined with *other* pressure indicators, like swap usage or allocator stalls.

Other times the miss is silent: a JVM that never evicts because it never grows, or a Redis instance with `maxmemory-policy noeviction` that just starts failing writes instead. No evictions, no warning, then a hard crash. That hurts. The checklist for these cases: confirm which cache layers actually *can* evict, verify your metrics agent isn’t sampling too coarsely, and check that your eviction counter resets after deployment — version bumps will skew baselines for hours.

“The absence of evictions is not the absence of risk. It might mean your system gave up silently instead of signaling.”

— paraphrased from a production engineer’s post-incident note

Debugging a misread eviction pattern follows a tight loop: pull the reclaim stack traces, correlate with allocator metrics, and check whether page-cache or slab memory dominated the reclaim. If the trace shows `shrink_inactive_list` running hot while `shrink_slab` is idle, that’s anonymous memory pressure — think app heap. If the opposite, it’s dentries or inodes — think file handles, log rotation, or a directory scan. Wrong order. Not yet. Those two distinctions alone will cut your false-positive rate in half.

Eviction Pattern FAQ and a Quick Pre-Outage Checklist

How many evictions before I should worry?

There is no magic number, and anyone who tells you otherwise is selling a dashboard. I have seen clusters fail after a dozen evictions and hum along after ten thousand. The count only matters when you know your baseline—your normal rate of churn under steady load. If your cache evicts 200 objects per minute on a quiet Tuesday, a jump to 400 is noise. If it jumps to 2,000, you're watching a seam split. Track the rate of change, not the raw total. The real warning is acceleration: evictions climbing faster than your traffic, or evictions that persist after the load spike has passed.

The second question people ask is whether the type of eviction matters. It does—enormously. A least-recently-used eviction on a warm cache is a shrug. A forced eviction from a memory-pressure signal means your allocator is fighting the kernel for pages. That second one is the one that precedes the outage. Watch for evictions that come in waves, not a steady trickle. Waves mean something is oscillating—usually a resize loop or a neighbor tenant slurping shared memory. That hurts.

Can I automate a response to eviction cycles?

Yes, but carefully. I have watched teams wire an alert to a cache-flush script and then spend a Friday afternoon explaining why they deleted half their session state. Automation without a guardrail is just a faster way to shoot yourself. The safe pattern is tiered: log everything, alert on acceleration, page a human when the acceleration crosses your pain threshold. Only then, if you have a rule that survived a postmortem, consider an automated action—and make it reversible. Evicting the evictor, so to speak, means you know what you're protecting and what you're willing to lose.

The trade-off here is latency vs. safety. A fully automatic response can shave seconds off your recovery time, but it can also mask the underlying cause. The catch is that if you automate away the symptom, you might never fix the disease. I would rather have a noisy page at 2 a.m. than a silent cascade that empties my primary cache and then thrashes the database. Wrong order is worse than no order.

Checklist: five things to review before your next shift

Most teams skip this, but a five-minute pre-shift scan beats a two-hour incident call. First, check your eviction rate trend over the last 24 hours—is it flat, climbing, or sawtooth? Second, look at the eviction reason breakdown; if "memory pressure" is climbing while "LRU" is flat, that's your red flag. Third, verify your cache hit ratio against your baseline; a 5% dip that persists is worth a conversation.

Fourth, inspect your largest cache keys. I have seen a single session object—bloated with user preferences and a decade of cruft—account for 30% of evictions because it kept getting ejected and rebuilt. That's a design bug wearing an eviction costume. Fifth, confirm your alert thresholds actually match your current capacity. A threshold tuned for a 4GB cache is noise when you're running 16GB—or worse, silent when you're down to 2GB. Adjust them, or you will chase ghosts.

Evictions are not the outage. They're the sound of the system clearing its throat before it screams.

— field note, after a cache-related production incident

The last thing to do is simple: pick one metric—eviction rate, hit ratio, or memory pressure—and write down what a "warning" looks like for your system. Not a generic threshold, your actual number. Share it with your team. Then go home. The pattern will still be waiting tomorrow, but now you will recognize it.