When Throughput Benchmarks Ignore Eviction Patterns: A Qualitative Redis Check

I once watched a team proudly present a benchmark: 150,000 SETs/sec on Redis 6, zero evictions. The CTO nodded. Then production hit. Evictions started, latency doubled, and the same benchmark suddenly showed 50,000 ops/sec. The problem wasn't Redis—it was how they'd measured throughput without accounting for eviction patterns.

Most throughput benchmarks treat Redis like an infinite bucket. But memory isn't infinite. The moment eviction kicks in, performance curves become jagged. This article is a qualitative field guide for engineers who've seen that jagged line and wondered what to fix first.

Where Eviction Blind Spots Show Up in Real Workloads

Benchmark environments vs. production memory pressure

I once watched a team celebrate a throughput test — 98,000 ops per second, Redis barely sweating. Two weeks later, same service cratered at 12,000 ops under production load. The difference? The benchmark ran on a warm cache with infinite headroom. Production ran at 87% maxmemory with a volatile TTL workload. That gap isn't a configuration tweak — it's a fundamental blind spot in how most teams design throughput benchmarks. You can push 100k writes per second into an empty Redis instance all day. The moment eviction kicks in, every single operation pays a tax nobody measured during staging.

'The benchmark said we were fine. Production said otherwise. The only variable was whether keys had to die to make room.'

— SRE lead, post-mortem on a cache-layer outage

The tricky bit is how eviction latency scales. Under low memory pressure — say, 50% used — a key insertion is nearly O(1). You allocate, you write, you move on. Cross the eviction threshold and suddenly the same insert triggers an LRU walk, candidate sampling, and a deletion. That cost is invisible in a benchmark that loads data once and reads repeatedly. But real workloads churn. Real workloads mix hot keys with one-hit-wonders. The eviction latency cliff hits when your benchmark assumes the allocator is the bottleneck. It isn't — the reaper is.

The eviction latency cliff nobody measures

Most throughput dashboards track p50 and p99 latency on the happy path. They don't instrument what happens when maxmemory-policy allkeys-lru activates mid-request. I have seen a Redis cluster sustain 5ms p99 for hours, then spike to 230ms within thirty seconds because a bulk load filled the last 200MB of headroom. That cliff is steep, and it's repeatable. The root cause is not network or CPU — it's the sampling cost of finding eviction candidates under pressure. Redis's volatile-TTL policy scans keys with expiry set; allkeys-lru samples a subset of the keyspace. Under heavy write load, the sample size may fail to find a good victim, forcing multiple rounds. Each round adds microseconds. Multiplied across thousands of concurrent connections, microseconds become milliseconds. Milliseconds become timeouts.

What usually breaks first is the application's retry logic. Clients see a slow response, reconnect, replay the write — which lands on the same overloaded node, accelerating the eviction cycle. The benchmark never tested that feedback loop. It assumed linear degradation. Real systems degrade in hockey-stick curves. And here is the uncomfortable truth: many throughput benchmarks published by vendors or open-source maintainers run on dedicated hardware with 30% memory utilization. They're not lying. They're simply measuring throughput in a world where eviction doesn't exist. That world doesn't match yours.

One pattern I have fixed repeatedly: teams benchmark with keyspace-events disabled, no TTLs set, and a dataset that fits entirely in memory. Then they deploy against production data where 40% of keys expire every hour. The eviction policy shifts from never invoked to every write triggers a death. The performance numbers don't just differ — they invert. Reads that were sub-millisecond now block behind eviction-linked compaction. The benchmark was not wrong. It was irrelevant. Ignore eviction patterns in your throughput tests, and you're not benchmarking Redis — you're benchmarking an empty room.

That hurts. Because the fix is not harder — it's just more honest: run your benchmark at 85% memory pressure, with real TTL distributions, and measure the cost of the eviction policy itself. Most teams skip this. They should not.

Foundations: What Most Teams Get Wrong About Eviction Policies

volatile-lru vs. allkeys-lru: not just a preference

Most teams pick a policy the way they pick a default font—quick glance, click, done. I have seen production playbooks where engineers set volatile-lru because “we set TTLs anyway” and never check what happens when those TTLs are missing. That's where the seam blows out. volatile-lru only evicts keys that carry an expiration—if your application code forgets to sprinkle TTL on 30% of writes, those keys become immortal tenants. The policy starves itself of eviction targets while memory fills with dead weight. Meanwhile allkeys-lru treats every key as evictable. Sounds aggressive—until your hotspot data starts vanishing under burst load. Wrong order. The real question is not “do we have TTL?” but “can we tolerate losing any key in the dataset?”

The catch is that allkeys-lru can mask sloppy key hygiene. I fixed a pipeline throughput regression once where the team blamed Redis cluster latency—turned out allkeys-lru was silently recycling recently-written session tokens that had no TTL but were still hot. Throughput looked stable in the benchmark because the eviction rate was constant. Constant eviction is not stable throughput. It's a controlled fire. The benchmark never measured which keys disappeared, only how many requests completed per second. That mismatch—request count versus data integrity—is where eviction blind spots hide.

Flag this for redis: shortcuts cost a day.

Why maxmemory-policy alone isn't enough

Setting maxmemory-policy and walking away is like installing a fuse box and never checking the wiring. A single policy directive can't compensate for access patterns that oscillate—bursty writes at 10:00 AM, then read-heavy traffic at 2:00 PM. I have seen benchmarks double throughput after lunch because the eviction daemon had been thrashing against cold keys during the morning spike. The policy was allkeys-lfu, which sounds robust. Honesty—it collapsed under a workload where frequency counts decayed faster than the eviction cycle could refresh. The result: hot keys got evicted because their frequency counters reset during a quiet minute, and throughput cratered.

What usually breaks first is the gap between the policy’s sampling behavior and real access locality. Redis samples a subset of keys (default 5) before evicting. That sample size is a control knob most teams ignore. Crank it to 10 and you increase eviction accuracy—but also shave microseconds off every eviction cycle. Under 100,000 ops/s, those microseconds compound. The trade-off is ugly: precise eviction steals throughput, and fast eviction evicts wrong keys. Most throughput benchmarks never stress this tension because they run flat-line workloads. Flat lines don't trigger eviction thrash. Real workloads do.

'We benchmarked with allkeys-lru and got 95th percentile latency under 2ms. Then we deployed. Eviction kicked in. Latency tripled. The policy didn't change—the access pattern did.'

— Senior engineer, post-incident review for a ad-serving pipeline

That anecdote cuts to the heart of it. Teams tune the policy variable while ignoring the interaction with maxmemory-samples, lfu-log-factor, and the cluster’s memory-to-request ratio. The result is a benchmark that passes but a system that bleeds. Fixing this means testing under eviction-dominant conditions—deliberately fill 85% of memory before firing the first throughput measurement. Not yet. Most performance labs fill to 50% and call it stress. That's a comfort zone, not a stress test.

Eviction-Aware Patterns That Actually Work

Keyspace sizing to stay under maxmemory

The most boring fix works best: keep your keyspace small enough that eviction never fires. I have seen teams run Redis at 70% maxmemory, sleep soundly, and still hit eviction storms during traffic spikes. That 30% buffer evaporates fast when a background job bulk-loads temporary keys. The trick is not just setting maxmemory—it's understanding your working set at peak, then carving a hard ceiling 15–20% below that number. One team I worked with cut their eviction rate from 12,000 keys per second to zero just by adding a second Redis instance for ephemeral cache data. Painful migration. Worth it.

But here is the uncomfortable truth: most throughput benchmarks never test with the keyspace at 95% of maxmemory. They run cold caches or tiny datasets. Then production arrives, memory pressure climbs, and the first eviction event annihilates p99 latency. A client once showed me a chart—flat sub-millisecond responses, then a jagged spike to 600ms the moment the evictor touched a hot key. The benchmark had lied because it never simulated the eviction ramp.

Using volatile-ttl with predictable TTLs

If you must run near the limit, volatile-ttl beats allkeys-lru for workloads where TTL discipline exists. This policy evicts the key with the shortest remaining TTL first. Sounds obvious. Most teams skip it because they set random TTLs—five minutes here, an hour there—and the evictor behaves erratically. The fix is normalization: assign TTLs in strict tiers. A session cache gets 300 seconds. A product lookup gets 900. A reference dataset gets 3600. No drift, no surprises. When the evictor scans, it picks the right key every time.

The catch is that volatile-ttl requires every key to have a TTL. Miss one—just one persistent key—and the policy degrades. I debugged a system where a single config object had no TTL, lived forever, and forced the evictor to cycle through thousands of younger keys needlessly. The fix? A startup script that audits all keys for missing TTLs and refuses to connect if any exist. Harsh. But the eviction rate dropped 80% overnight.

'Eviction is not a failure state—it's a design parameter you chose not to plan for.'

— paraphrased from a production incident post-mortem I sat through, 2023

What usually breaks first is the interaction between eviction policy and key access patterns. A team using allkeys-lfu assumed frequently accessed keys would stay. They didn't account for a batch job that read millions of cold keys once—those one-hit-wonders polluted the LFU counters and evicted hot keys. The fix was splitting the batch workload into a separate Redis instance with allkeys-random. Different workload, different policy. That's the pattern that actually works: isolate access patterns, then tailor eviction per instance.

Anti-Patterns That Trigger Reverts

Believing 'More Memory' Fixes Everything

I watched a team double their Redis instance size last quarter. Throughput benchmarks screamed—30% higher ops/sec, latency flat, everyone high-fived. Two weeks later they reverted. The problem wasn't memory pressure; it was eviction behavior. They’d cranked maxmemory from 8GB to 16GB but left the policy on allkeys-lru. That sounds fine until you realize their workload was a write-heavy session cache with TTLs scattered across a 24-hour window. The extra headroom just delayed the inevitable—when memory filled again, Redis evicted old sessions en masse. Users got logged out. Support tickets exploded. The revert came with a postmortem titled "We fixed the wrong number."

Field note: redis plans crack at handoff.

The mistake is seductive because it works—briefly. More RAM defuses eviction pressure, but only if your eviction policy actually matches what you cache. For session data, volatile-ttl or volatile-lru usually beats allkeys-lru because you control which keys expire. Without that alignment, you're just renting a bigger parking lot for a car that shouldn't be parked there. I have fixed this exact scenario three times this year. Each time, the team's throughput benchmark had been run on a cold cache or under uniform key-access patterns—real-world bursts killed them. One engineer told me, "Our staging tests never showed this." Of course they didn't—staging had 4GB of data and no eviction pressure. Production had 14GB and a ticking clock.

The Noeviction Trap in Caching Layers

noeviction is a promise: "I will never drop data." In a pure database, that's noble. In a caching layer, it's self-sabotage. Teams slap it on because they fear losing hot keys, and then throughput collapses when OOM errors cascade through their app. I saw a Redis cluster serving API response cache—noeviction, 12GB max, healthy hit rate. Then a promotion hit. Traffic spiked 4x, cache filled to the brim, and every new SET returned an error. The app didn't degrade gracefully—it crashed. The revert? Swap to allkeys-lru and lose 15% of cached responses. But that 15% loss was far cheaper than four hours of downtime.

The catch is cultural: engineers treat eviction as failure rather than a design parameter. noeviction feels safe; it feels like control. But caching is about probability, not perfection. You want the system to shed cold data automatically—that's good. When throughput benchmarks ignore eviction entirely, teams optimize for a vacuum. Then production hits, eviction fires, and the revert rips out that "optimization" in an afternoon.

'We spent three sprints tuning pipeline batching. One noeviction config undid it in ten minutes.'

— Senior SRE, post-mortem chat

What usually breaks first is not the eviction itself, but the monitoring gap. Nobody watches eviction rates. They watch latency, CPU, hit ratio—never the steady drumbeat of keys being ejected under the default policy. That drumbeat becomes a roar when workload shifts. Want a concrete next action? Add evicted_keys to your throughput test dashboards and set a warning at 10 evictions/sec. If that number jumps when you scale memory, the policy is wrong—revert before the users do.

Long-Term Costs: Eviction Drift and Maintenance Burden

How eviction behavior changes across Redis versions

You deploy a stable configuration. Six months later, a version bump quietly rewrites your eviction guarantees. I have watched teams pin Redis 6.2.6 for eighteen months because upgrading to 7.0 flipped their maxmemory-policy from allkeys-lru to volatile-lfu — without any config change, just a shifted default in the upgrade notes nobody read. The seam blows out at 2 AM when a cache-heavy microservice starts evicting session keys instead of cold artifact blobs. That hurts. Redis 7.2 introduced lazy-free eviction by default; suddenly your eviction latency dropped by 40%, but the eviction order itself changed because the background thread reaps differently under memory pressure. Most teams skip this: what worked under synchronous eviction logic may fail under asynchronous freeing — keys that should have been evicted first linger long enough to serve stale data. The catch is that these behavioral shifts are rarely called out in changelogs as performance-critical. They're called bug fixes or minor improvements. Your throughput benchmark from last quarter is now a historical artifact, not a reference point.

The hidden cost of lazy free on eviction latency

Lazy free sounded like a gift. Free memory asynchronously, keep the event loop responsive, never block on big key deletion. But here is the pitfall: lazy free doesn't eliminate eviction latency — it hides it behind background threads that accumulate debt. Imagine you're running allkeys-lfu with a maxmemory limit of 4 GB. Your write throughput spikes, the eviction daemon fires, lazy free queues 200 large hashes for background deletion. The event loop stays green — throughput looks fine on your dashboard. Meanwhile, the background reaper falls behind, memory stays allocated longer than the policy intended, and the next wave of writes triggers more evictions because the free space hasn't materialized yet. You lose a day debugging why TPS drops cyclically every 90 seconds. The root cause? Lazy free introduced a two-phase eviction that your benchmark never measured: the eviction decision happens instantly, but the actual memory recovery drifts by hundreds of milliseconds per cycle. Over eight hours, that drift accumulates into a maintenance burden — your team writes cron jobs to drain the backlog, tunes lazy-free-threshold, and eventually questions whether the asynchrony was worth the complexity.

'Lazy free transformed eviction from a deterministic event into a probabilistic queue. The benchmark that measured latency in microseconds missed the drift that costs milliseconds in production.'

— Site reliability engineer, after three months of biweekly pager rotations tied to eviction stalls

A concrete situation: I once consulted for a team running 256 Redis nodes with maxmemory-policy set to allkeys-random and lazy-free enabled. Their throughput benchmarks showed 99th percentile latency under 2 ms. Beautiful numbers. But their maintenance burden was brutal — every two weeks a different node would hit 100% CPU on the background purging thread, dropping write throughput by 30% for minutes at a time. The eviction drift? They were freeing keys faster than the policy could evaluate which keys to evict next, creating a loop where the eviction runner and the free thread fought over the same memory regions. The fix was counterintuitive: disable lazy free entirely, accept the blocking eviction latency, and gain deterministic eviction order back. Their throughput dropped 5% but their pager volume dropped by 80%. That's the trade-off you never see in a three-hour benchmark run. The maintenance burden is not the eviction itself — it's the accumulated cost of diagnosing which version-specific behavior broke your assumptions, retuning thresholds, and writing yet another alert rule. Over two years, that cost exceeds any latency gain lazy free provided. What usually breaks first is your team's tolerance for drift, not the database.

When Eviction-Based Designs Are the Wrong Call

Persistent Data Stores That Can't Tolerate Eviction

I once watched a team burn two weeks tuning Redis eviction policies for what they thought was a cache. It wasn't. Their service stored session tokens — lose one and a user gets kicked out mid-checkout. Benchmarks showed screaming throughput under volatile-LRU. Real traffic showed support tickets exploding. The mistake? They ran throughput tests with eviction enabled and assumed the 99th percentile latency looked fine. What they missed: every eviction was a data loss event, not a cache miss. When your store is the system of record, eviction isn't a performance lever — it's a bug.

Most teams skip this: run the same benchmark with noeviction and watch writes fail instead. That silence is terrifying. For persistent stores — user profiles, transaction logs, state machines — the benchmark must assert zero evictions, not just fast evictions. The throughput number doesn't matter if the data vanishes.

'Eviction benchmarks measure speed of forgetting. If forgetting is illegal, the benchmark is worse than useless.'

— overheard at a post-mortem for a dropped-order incident

The catch is subtle. Even with noeviction, Redis returns errors when memory fills. Your throughput test must include error-rate assertions, not just ops-per-second. I've seen dashboards showing 50k req/s — all returning OOM command not allowed. That's not throughput. That's a denial-of-service simulation in production clothing.

Flag this for redis: shortcuts cost a day.

Latency-Sensitive Workloads Where Eviction Spikes Are Unacceptable

Eviction isn't free. Every kicked key triggers a synchronous DEL or lazy free — both consume CPU cycles on the same event loop serving your requests. In a latency-sensitive pipeline — think ad serving, real-time bidding, or payment auth — an eviction wave can double P99 latency for hundreds of milliseconds. Your benchmark ran with steady-state memory at 70% capacity. Production memory hit 95% during a flash sale. The eviction rate jumped from 2 keys/second to 800 keys/second. That's the spike that kills the SLA.

The fix isn't better eviction. It's removing eviction from the critical path entirely. Two patterns I've seen work: pre-allocate a fixed-size pool with maxmemory-policy allkeys-lru and size it so evictions happen during off-peak windows only. Or — honestly — use a different store. When your workload can't tolerate a 300ms tail latency jitter from a lazy free storm, Redis with eviction is the wrong tool. The throughput benchmark that shows 100k ops with zero eviction cost is lying unless your memory headroom never shrinks.

What usually breaks first is the assumption that eviction is uniform. It isn't. A single large object evicted can stall the event loop longer than a hundred small ones. I've traced production incidents where an eviction of a 5MB JSON blob caused a 450ms pause — the benchmark never stored objects bigger than 1KB. Wrong order. Your latency SLA needs an eviction-free guarantee, not a fast-eviction guarantee. If you can't promise that, the pipeline benchmark is measuring a fantasy.

Open Questions: Sampling, LFU, and Policy Trade-Offs

Does sampling size impact eviction accuracy?

Most teams treat the sampling window as a dial you turn once and forget. I have watched engineering leads set it to the Redis default of five samples per eviction cycle and call it done. That works fine—until your working set balloons past memory limits and the eviction algorithm starts tossing keys it should have kept. The catch is statistical: five samples from a key space of two million entries gives you a comically small glimpse of the access distribution. A hot key that accounts for eighteen percent of reads has roughly a sixty-five percent chance of not appearing in any given five-element sample. Wrong order. You evict a needle, keep a haystack, and throughput collapses.

Bumping the sample size to ten or fifteen improves eviction accuracy measurably—but it also consumes CPU cycles per eviction call. We fixed a production incident where a team cranked samples to twenty-five and saw latency spike by twelve milliseconds during peak writes. The trade-off is brutal: more samples protect cache warmth but steal time from the request pipeline. Honest advice—test this with your actual access pattern, not a synthetic key generator. A 10-sample window doubled eviction precision on one of our read-heavy workloads while adding less than three percent CPU overhead. The same setting on a write-heavy cluster blew up tail latency by nineteen percent. There is no universal answer.

When does LFU outperform LRU in practice?

LRU makes a quiet assumption: the most recently used key will be needed again soon. That belief holds for session caches, rate-limit counters, and short-lived payloads. But consider a content catalog where a few items get hammered repeatedly while hundreds of less popular keys sit untouched for hours. LRU will happily evict that stale-but-once-popular entry the moment a new write arrives—even if the stale key is accessed twice a day. LFU, by contrast, tracks frequency. The old reliable key survives eviction because its access count is high, while one-hit wonders get dropped first.

'We switched to LFU expecting a magic bullet. Instead, our cold-start latency tripled because new keys couldn't build frequency fast enough.'

— Lead SRE, mid-size ad-tech platform

The pitfall is visible in any workload with bursty traffic. A key that receives two hundred requests in one minute then zero for six hours accumulates a high frequency score. That score persists, keeping the key alive while newer, timelier entries get evicted. I have seen this pattern produce stale dashboards that showed data four hours old because LFU refused to let go of yesterday's top article. The workaround—decay counters or periodic frequency halving—adds complexity and another knob to misconfigure.

So when does LFU actually win? Workloads with a long-tailed access distribution where a small set of keys dominates traffic over days, not minutes. Think analytics aggregators, reference data sets, or user-profile stores where the same ten thousand records serve ninety percent of reads. LRU would cycle through those records every time a new batch of ephemeral writes arrives. LFU holds the hot set stable. That stability reduces eviction churn and keeps pipeline throughput flat. Not flashy. But flat throughput under load beats a sawtooth pattern every time.

Summary: What to Fix in Your Next Throughput Test

Actionable checklist for eviction-aware benchmarking

Most throughput tests lie. Not on purpose — but they lie because they test a warm cache that never faces real eviction pressure. Your next benchmark needs to break that illusion. Start by running a baseline that measures peak throughput with zero evictions. Then introduce a second phase where you cap memory at 60% and flood keys until the eviction daemon is constantly firing. The delta between those two numbers? That's your real budget — everything else is wishful thinking. I have seen teams celebrate 200K ops/sec, only to watch their production pipeline crater at 40K when the LRU threshold kicks in. The catch is simple: if your benchmark doesn't throttle during eviction storms, it doesn't reflect reality. Fix that by adding a third phase — one that alternates burst writes with idle reads — exactly the pattern that exposes TTL-based or LFU-based collapse.

Next experiments to run

Stand up a single-node test instance. Load it with 10 million small objects under two policies: allkeys-lfu and volatile-ttl. Then run a three-minute workload that does 80% reads and 20% writes — but every tenth write sets a TTL of one second. Benchmark throughput. Then benchmark again with TTLs set to one millisecond. What breaks first? The eviction path quickly becomes the bottleneck when expiration rates exceed the sample window — a detail most published benchmarks ignore entirely. That said, don't fall for the trap of adding bigger instances. More memory just delays the collision; it doesn't change the policy's behavior under load. One team I know doubled their cluster size and saw throughput drop because the eviction sampling became proportionally slower on larger key spaces. Wrong order of operations.

Maybe you don't yet see the drift. Run the same experiment daily for a week — log the eviction count per second. It will climb, not stabilize, as stale keys accumulate and the policy starts cycling live data. That incremental creep is the maintenance burden nobody budgets for.

“My cache hit rate was 98% in staging. In production it oscillated between 60% and 75% — the benchmark never told me why.”

— Senior engineer at an ad-tech company, after replacing allkeys-lfu with an eviction-aware hybrid

One more thing: replicate your exact production eviction count, not just the memory limit. If production evicts 5K keys per second but your test evicts 200 — the eviction pipeline is not stressed. Adjust maxmemory-policy parameters until the eviction rate matches. That hurts to set up. Do it anyway.