When Adding Nodes Fails to Improve Throughput: A Topology Check

You add three more Redis nodes to your cluster. yield should climb, right? Instead, latency spikes, error rates jump, and clients start timing out. Sound familiar?

More nodes don't automatically mean more output. In fact, in a Redis Cluster, topology—how nodes are arranged, how slots are distributed, and how clients route requests—often matters more than raw shard count. This article lays out exactly what to check when scaling fails to deliver. No theory for theory's sake; just the specific topology issues I've seen kill performance in production, and how to spot them before you add another node.

Who This Topology Check Matters To

The DevOps engineer staring at a flat volume graph after scaling from 6 to 12 nodes

You doubled your Redis cluster nodes—patted yourself on the back, maybe even bragged in stand-up. Then the metrics came in. yield flatlined. Worse? Latency tails stretched. I have seen this exact graph on a client dashboard, and the first instinct is always hardware: bigger instances, faster SSDs, more network bandwidth. That's almost never the fix. The topology is the fix—or rather, the absence of a topology check is the trap. What usually breaks first is the mapping between data and nodes, not the raw compute. Adding nodes without redistributing slot assignments is like hiring more cashiers but never telling customers which register to queue at.

The catch is subtle: Redis Cluster uses 16,384 hash slots, and your client library maps keys to those slots using a local copy of the cluster's topology. If that map is stale—or worse, never updated after a resharding operation—requests pile onto the same few nodes. The new nodes sit idle. You lose a day chasing CPU credits when the real problem is a misaligned slot map. I fixed one case where a team had seven idle replicas because their redis-py-cluster client cached an old node list. Seven. Idle. That hurts.

The architect designing a multi-AZ Redis Cluster on AWS or GCP

Multi-AZ sounds bulletproof until you watch output crumble during a failover. Here is the scenario: you have three AZs, six nodes, traffic splits evenly—until the primary in AZ-b dies. The new primary, promoted in AZ-c, inherits the same slot range but sits behind a different network path. Your client? It still sends some write traffic to the old AZ-b endpoint, because the client's topology table only refreshes on MOVED errors. Those errors pile up. Retries spike. output graph goes diagonal—downward. Most groups skip this: verifying that client-side slot maps actually converge after a topology change. They test the cluster health, not the client's perception of the cluster. flawed order.

Cloud adds another layer: cross-AZ latency variance. If your slot distribution scatters related keys across AZs, every multi-key operation pays a cross-region penalty. I have seen architects put all hot keys into a one-off AZ to avoid that cost—then wonder why scaling from 6 to 12 nodes didn't help. The bottleneck was AZ-local bandwidth, not cluster capacity. The topology check they skipped? Identifying which slots held hot keys and whether those slots were geographically concentrated. Not yet on your radar? It should be.

Anyone using client-side routing without verifying slot maps

Client-side routing libraries like Lettuce, JedisCluster, or redis-py-cluster are efficient—until they aren't. They cache the cluster topology locally, then route requests based on that snapshot. The problem: that snapshot can be off for minutes after a topology change. Worse, some libraries only refresh the map after encountering a MOVED redirect. If your traffic pattern never hits the exact key that triggers a redirect—say, you only access a subset of keys—the stale map persists indefinitely.

Honestly—I debugged a production incident where a sharded session store sent all write traffic to two nodes, even though the cluster had eight healthy primaries. The client library had learned the topology at startup and never rechecked. The fix wasn't more nodes. It was forcing a topology refresh and adding a health-check loop that validated slot mappings against the live cluster state.

'Scaled nodes, zero gain. Checked slot maps. Found client routing to dead primaries for 70% of keys. Fixed the map. yield tripled in ten minutes.'

— SRE lead, post-mortem on a misrouted Redis Cluster

Flag this for redis: shortcuts cost a day.

Flag this for redis: shortcuts cost a day.

If you use direct client access (no proxy), you own the topology sync. That means you also own the failure when it drifts. A proxy like Envoy or HAProxy can hide some of this—but proxies add latency and become bottlenecks themselves. The trade-off is real: direct routing gives you lower latency if your maps are correct, but one stale map erases that advantage instantly. The next section will show you exactly what to check before you blame the hardware. For now, ask yourself: when was the last time you verified that your client's slot map matched the cluster's actual topology?

What You Need Before Running the Check

Access to the Cluster: CLUSTER INFO, CLUSTER NODES, CLUSTER SLOTS

You can't diagnose what you can't see. Before you run any yield test, pull these three commands from a Redis CLI client connected to any node in the cluster. No GUI shortcuts—raw terminal output. CLUSTER INFO gives you the cluster_state (should be ok), the number of nodes, and the crucial cluster_slots_assigned count. If that number is less than 16384, you already found a gap. CLUSTER NODES dumps every node ID, its IP:port, its role (master or replica), and—this is the part most people skip—the connected flag and the pings_sent/pings_received counters. A node that shows disconnected or a rising ping imbalance is a red flag, not a dead node. Finally, CLUSTER SLOTS maps every slot range to a master. Cross-check that against your application's hash tags. I have seen crews add three nodes and then discover that their entire write load still hit the same 2,000 slots—because the new nodes were empty. That hurts.

Client-Side Metrics: Command Latency, Redirected Requests, MOVED/ASK Errors

The server looks pristine. The client, though—that's where the rot hides. Pull per-node latency from your client library's histogram, not just averages. P99 spikes above 50ms after a scaling event usually mean the client is hammering a stale topology table. Track redirected requests next. Every time a client sends a key to the off node, Redis replies with a MOVED or ASK error—then the client re-issues the command to the correct node. That doubles latency for that lone operation. A 1% redirect rate might not feel like much, but spot-check a busy node: if your redirected requests exceed 100 per second on a 5,000 ops/s node, your client-side slot map is stale. The fix is not a cluster restart—it's forcing a refresh of the CLUSTER SLOTS cache on every client connection. Most drivers have a cluster-refresh config knob. Use it. The catch is that some proxies (like Twemproxy) mask these errors by batching, so you'll see lower errors but higher tail latency—a trade-off that fools many.

“We checked node CPU, disk, and memory. All looked fine. yield was still flat after adding four nodes. Turned out the client was still sending 80% of writes to the original two masters.”

— Redis ops lead, after a post-mortem that delayed a product launch by two weeks

A Baseline: output Per Node Before the Last Scaling Event

Most crews skip this. flawed order. Without a per-node yield baseline recorded before you added nodes, you're guessing. Not educated guessing—pure hopium. Capture three metrics for each master: instantaneous_ops_per_sec (from INFO stats), network input bytes per second (total_net_input_bytes delta over 60 seconds), and the number of connected clients. Store that snapshot with a timestamp. Why does this matter? Because after scaling, if the new nodes show ops/sec at 10% of the old nodes' baseline, your hash slots are unbalanced. Not yet dangerous—but it explains the flat yield. Conversely, if the new nodes match the old baseline but overall throughput didn't budge, your bottleneck shifted from CPU to network bandwidth or to a shared resource like a Replication Backlog on a single replica. I have fixed exactly this by reading the baseline and realizing the cluster wasn't CPU-bound—it was memory-bandwidth-bound, and adding nodes just spread the same memory strain across more machines. The baseline catches that lie. Take the snapshot. Keep it. You will need it in forty-eight hours when someone asks, "Did the scaling actually help?"

Step-by-Step Topology Audit

Check slot distribution: any node holding >10% more slots than others?

Open redis-cli --cluster check 127.0.0.1:6379 and watch the slot counts per master. I have seen clusters where one node carried 2,800 slots while others sat at 1,700. That asymmetry alone throttles throughput—the overloaded node becomes a serial bottleneck for any hash-slot-heavy write pattern. Reshard with redis-cli --cluster rebalance, but understand the trade-off: rebalancing moves keys live, which spikes latency for the duration of the move. Do this during a maintenance window, not at peak traffic. The rule of thumb? If any node holds more than 10% extra slots relative to the cluster average, you're leaving throughput on the table. flawed order. Fix the imbalance before blaming the network.

Verify client slot cache: stale maps cause cross-slot redirections

Most groups skip this. Your application holds a local map of slots to nodes—what happens when the topology changes? A reshard or a failover invalidates that map, yet many clients keep firing requests at the old owner. Redis returns a MOVED error, the client reconnects, and you lose a full round-trip per command. The seam blows out under pipeline load. Test this: force a manual reshard, then watch your client logs for MOVED responses. If you see more than a handful per second, your client’s slot refresh logic is too lazy. Some drivers refresh on every MOVED; others batch updates. Pick the batch variant. Stale caches hurt more at scale—one bad client can amplify cross-slot redirections across hundreds of connections.

Measure cross-slot multi-key operations: they kill parallelism

Here is the concrete test. Fire a pipeline of single-key SET commands—measure the throughput. Then fire the same number of multi-key MGET commands that span two slots. Throughput drops. That hurts. Every cross-slot operation forces the proxy (or your client) to scatter requests across nodes and coordinate responses. The catch is that many applications treat Redis as a simple key-value store and unknowingly scatter keys across slots. Run redis-cli --cluster info 127.0.0.1:6379 | grep slot to see your key distribution, then sample your application’s multi-key call patterns. If more than 5% of your multi-key operations cross slots, you have a topology-induced bottleneck. Redesign your key hashing—use hash tags: {user:42}:profile forces all keys under one slot. That trades better parallelism for tighter hash distribution, but in practice the throughput gain outweighs the cluster imbalance risk.

“We ran this audit on a 6-node cluster. Resharding + hash tags doubled our pipeline throughput. The topology was the problem, not the nodes.”

— engineer at a mid-size ad platform, after chasing network bottlenecks for two weeks

Field note: redis plans crack at handoff.

Field note: redis plans crack at handoff.

Test with a single-threaded client vs. pipelined to isolate topology overhead

Run two benchmarks against the same cluster. First, a single-threaded client issuing one command at a time—sequential, no pipelining. Record the baseline. Second, the same client with a pipeline depth of 50. Compare the throughput ratio. If the pipelined run shows less than a 10x gain over the sequential run, your topology is eating the pipeline benefit. The culprit is usually cross-slot redirections or uneven load distribution forcing the pipeline to wait on the slowest node. A clean cluster should give you a 15x to 20x boost under moderate pipeline depth. Anything worse means the audit found something—go back and check slot balance and client cache freshness. One rhetorical question: why optimize query logic when the cluster itself is miswired? Fix the seam first.

Tools and Commands You'll Actually Use

redis-cli — the two commands that tell the truth

Most crews skip this: they run redis-cli --cluster check 127.0.0.1:6379 and call it done. That output shows slot coverage and basic health—fine for a greenfield cluster. What it won't tell you is whether your hash slots are actually balanced across the hardware you added. I have seen a 12-node cluster where three nodes carried 74% of the slots after a resize. The check passed; throughput flatlined. That's where --cluster rebalance saves you—but only if you run it with --threshold 5 and --pipeline 100. Without those flags, the rebalance may refuse to move anything because the imbalance didn't trip the default 10% threshold. flawed order. You lose a day debugging why the new nodes sit idle.

The catch is that rebalancing is not free. It moves slot ownership, which triggers cluster-wide redirects—so schedule it during low traffic, or your redis_cluster_redirected_total metric will spike like a seismograph during a quake. Most groups skip this until prod hurts.

CLUSTER SLOTS output — parsing the mess

CLUSTER SLOTS dumps a raw array of slot ranges mapped to node IPs. Raw is generous—it looks like line noise. You need jq or a ten-line Python script to transform it into something human (or alertable). Our standard filter at rushcorex.top: redis-cli cluster slots | jq 'group_by(.[2][0]) | map({node: .[0][2][0], slots: [.[][0:2]] | unique | length}) | sort_by(.slots)'. That gives you a sorted list: which node owns how many slots. The first entry is your poorest performer. The last is the one you likely just added. If the ratio between max and min exceeds 1.3x, your topology is lopsided—even if --cluster check says green. I fixed a client's throughput stall exactly this way: a four-line script exposed that two nodes carried 6,100 slots each while a third carried 1,200. The cluster was technically "healthy" but operationally broken.

One rhetorical question: did your monitoring catch that before users complained? Ours didn't.

Prometheus — the silent witness

Prometheus exposes redis_cluster_slots_balanced if you use the Redis exporter (v1.53+). This metric is a boolean: 1 means balanced, 0 means not. Simple? Yes. Misleading? Also yes. The exporter defines "balanced" as each node holding within 10% of the average slot count. That sounds fine until you have a heterogenous cluster—say, 8GB RAM on old nodes and 32GB on new nodes. Slot balance doesn't equal resource balance. You need redis_cluster_redirected_total (rate over 5 minutes) and redis_cluster_known_nodes side-by-side. A sudden jump in redirected requests after a node addition means your clients are still talking to the old topology. We fixed this by forcing clients to refresh their cluster slots map: redis-cli -p 6379 CLUSTER RESET on the application side, followed by a soft reconnect. That hurts—but less than 500ms of timeout per request.

Custom scripts — the difference between knowing and guessing

Tools alone won't save you if you run them blindly. Write a script that compares CLUSTER SLOTS before and after a topology change—store the hash in a text file. Tomorrow, when throughput drops, diff the two files. If they match, the problem is not topology; look at network, keyspace, or a single hot key. If they differ, your cluster reconfigured itself (or someone ran reshard at 3 PM). We built a cron job that emails the team when the slot distribution delta exceeds 5%. It has caught three unplanned reshardings in two years. Worth the thirty minutes to write it.

“We thought adding nodes always helps. It doesn’t. We thought tools like redis-cli catch everything. They don’t.”

— CTO at a mid-size ad-tech firm, after two weekends debugging a flatlined cluster

Your next step: run redis-cli --cluster rebalance with a 5% threshold tonight. Then diff your slot map tomorrow morning. If nothing moved, you have a different problem. If something did, you just fixed a throughput bottleneck you didn't know existed. That's the only validation that matters.

Flag this for redis: shortcuts cost a day.

Flag this for redis: shortcuts cost a day.

When One-Size-Fits-All Doesn't: Variations for Proxy vs. Direct Access

Proxy architectures (Twemproxy, Envoy): topology hidden but resharding costly

You sit behind a proxy for a reason—clean client code, simpler failover, maybe some legacy routing you’d rather not touch. The proxy owns the topology map, and your app sends every SET and GET to a single endpoint that looks like one big Redis. That sounds fine until a reshard happens. I have seen groups add three nodes to a Twemproxy-backed cluster, watch the proxy reload its config, and then—nothing. Throughput flatlined. Why? The proxy hadn’t actually spread the new hash slots evenly. It held onto the old distribution, so the fresh nodes sat almost empty while the original ones kept fighting over hot keys. The proxy’s topology is a black box; you can't run CLUSTER SLOTS or redis-cli --cluster check because the proxy doesn’t expose those commands. Your only audit option is to monitor per-backend connection stats. If one node handles 70 % of traffic post-resize and the new nodes hover below 5 %, you have a proxy-side slot assignment problem, not a Redis one. The fix often means rebalancing the proxy’s hash-ring manually or, worse, flushing the whole mapping and re-seeding it. That hurts—downtime or a careful blue-green cutover.

Direct-access clients: stale slot maps are the main killer

Lettuce, redis-py-cluster, Jedis—these clients fetch the slot map once at startup, cache it locally, and move on. Most crews skip this: the map can become beautifully obsolete. A new node joins, slots migrate, and your client still thinks key user:4432 lives on old-node-5. It sends the request there, gets a MOVED redirect, and retries. One redirect costs maybe 2 ms. But when 20 % of your keys have moved, the retry storm doubles latency and burns CPU on both sides. The topology check for direct-access clients is not about the cluster itself—it's about the client’s view of the cluster. Run CLUSTER NODES directly and compare it against your app’s cached slot table (most drivers have a debug endpoint for this). A mismatch over five seconds old is a red flag. The fix? Force a refresh—either by calling CLUSTER RESET soft or by restarting the client pool in a rolling fashion. Annoying, yes. But I have fixed two separate production outages with exactly that move.

‘The proxy doesn’t care what the cluster knows. The cluster doesn’t care what the client cached. Your job is to make them agree—or pay the latency tax.’

— field note from a migration gone faulty at 2 AM (the author’s own)

Multi-region clusters: latency asymmetry from far-away replicas

Throw replicas across three AWS regions and you get a topology that looks fine on paper but hurts in practice. Each region has a primary and two replicas, slot ownership is balanced, no errors in CLUSTER INFO. Yet read latency from the secondary region spikes by 80 ms. The topology check you need here is not about slots—it's about replica placement and read-preference settings. Most drivers default to NEAREST replica, but NEAREST is a best-effort DNS guess, not a latency-aware election. Worse, if your client library uses a static replica list from the startup slot map, it may send reads to a replica 500 km away even when a local one exists. What usually breaks first is the cross-region REPLICAOF chain: the far replica’s replication lag grows during bursts, the client sees stale data, retries on the primary, and the primary’s CPU spikes from the extra load. Check INFO REPLICATION on each region’s nodes simultaneously. If the lag on a local replica exceeds 100 ms while the cross-region lag stays under 10 ms, your client is ignoring the local copy entirely. The variation for multi-datacenter setups is simple: audit the driver’s replica-selection logic, not just the cluster’s health. Add a LATENCY histogram per region to your metrics dashboard—if the histogram shows two distinct latency peaks, your topology is working against you. Fix the read-preference config, or shuffle replicas so that each region’s app talks only to its own zone’s copy. That alone can drop p95 latency by 40 %.

What to Check When Throughput Still Doesn't Improve

The hidden cost of MOVED redirections: measure redirect rate per second

Your cluster passes the audit—slots are assigned, nodes are reachable, every cluster info line glows green. Yet throughput is a flat line. I have seen crews blame the application layer for hours before someone runs redis-cli --latency-history and spots the real culprit: MOVED redirections eating the pipeline alive. Every time a client sends a command to the off node, Redis replies with -MOVED, the client re-issues the request, and the round-trip cost doubles. With a cluster of 12 nodes and a poorly configured hash slot map, you can accumulate 2,000–5,000 MOVED responses per second. That's not a cluster—that's a telephone game.

How do you catch it without staring at logs? Run redis-cli --stat on any node and watch the keyspace_hits versus keyspace_misses ratio, but more importantly, inspect cluster_stats_messages_sent and cluster_stats_messages_received from INFO. A high delta between those two counters often correlates with redirect storms. The fix is rarely a code change—it's a client reconfiguration to use CLUSTER NODES output to rebuild the local slot map. One engineer I worked with hardcoded the initial node list in a config file; after resharding, half the traffic went to the faulty partitions. We fixed it by bootstrapping from a DNS round-robin that returned all masters. That alone dropped redirects by 97%.

Slot imbalance after resharding: automatic rebalancing often leaves nodes uneven

You resharded. You ran redis-cli --cluster rebalance. Everything looks balanced on paper—each node shows roughly the same number of slots. But paper lies. The numbers I care about are used_memory per node and, more critically, the per-slot key distribution. A slot can hold 10 keys or 10 million keys, and the default rebalancer moves slots, not keys. The catch is that your three hot slots might each carry 8 million keys while the rest carry a few hundred thousand. That imbalance means three nodes handle 80% of the write traffic; the other nine are idle.

Most teams skip this: check redis-cli --cluster check localhost:7000 and look at the keys column, not just slots. If you see a node holding 15 million keys and another holding 600,000, the slot distribution is lying to you. The real solution is brutal: flush the hot slots and pre-shard your key space with hash tags that force an even spread. {user}:1000 over user:1000—a single curly brace changes everything. That said, retrofitting hash tags into a production schema is painful. I've done it on a Friday afternoon. Don't do it on a Friday afternoon.

“Slots are not bags of sand—they're icebergs. What you see above the water (count) tells you nothing about the mass below (keys).”

— veteran Redis admin, during a 3 AM incident call that could have been avoided

Network topology: cross-AZ latency on slot migrations can stall pipelines

Your Redis nodes span two availability zones. Good for fault tolerance—bad for throughput if you ignore the latency tax. During slot migration, Redis moves keys from source to target node synchronously. If those two nodes are in different AZs with a 2ms round-trip delay, every migrated key costs you those 2ms plus the local work. Multiply by 500,000 keys and a migration that should take 15 seconds now takes 20 minutes. Worse—clients that pipeline commands to the source node during migration get -ASK redirections, and -ASK is worse than -MOVED because the client must query a different node without updating its own slot table. The seam blows out.

What usually breaks first is the pipeline stall. A client sends a batch of 50 commands; 12 of them hit migrating slots. Those 12 get -ASK replies, the client serializes the remaining 38 while it fetches from the target, and suddenly your throughput graph looks like a sawtooth wave. I have measured this: a 1ms cross-AZ latency, during a slot migration on a 20-node cluster, dropped peak throughput from 80,000 ops/s to 32,000 ops/s. The fix is architectural—either colocate all masters in the same AZ (and accept the failure domain risk) or schedule migrations during low-traffic windows and cap migration speed with CLUSTER SETSLOT ... MIGRATING ... rate limiters. Don't let Redis run migrations at full throttle across AZ boundaries. It will hurt. Then it will hurt again during rebalancing next week.

One final thing to verify: redis-cli --cluster check won't show you cross-AZ mapping. You need cluster nodes output cross-referenced with your cloud provider's instance metadata. If the latency between two masters is >1ms on a 10 Gb link, you're paying for a topology mistake. Fix the topology—don't throw more nodes at it. More nodes in the wrong AZ just make the problem more expensive.