redis / redis/node-redis

cluster slots not rediscovered on scale-down cluster

Open
#2,806 5 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug
Dominant language
TypeScript
Stars
17.6k
Forks
2k
Avg merge
2d 3h
Merged PRs (30d)
40

Description

Description

#slots.rediscover() is correctly called when MOVED or ASK reply is received:

https://github.com/redis/node-redis/blob/6f79b49f731a1aaf57f42e16ea72774f062e57a1/packages/client/lib/cluster/index.ts#L271

https://github.com/redis/node-redis/blob/6f79b49f731a1aaf57f42e16ea72774f062e57a1/packages/client/lib/cluster/index.ts#L256

there is a problem / race-condition on scale-down cluster (rebalance hash slots away, forget node, shutdown):

#slots.rediscover() might never be called, if the leaving node has not been queried (after it's slots migrated away) before shutdown (hence it never got a chance to reply with MOVED and trigger a rediscover)

this results in connection closed errors, because node-redis cluster.#slots is keeping a client active for an outdated cluster topology.

steps to reproduce:

  1. start a redis cluster
  2. create node-redis client and connect
  3. redis-cli reshard all slots away from a redis cluster node
  4. redis-cli del-node the now empty node, then shutdown
  5. node-redis client execute command for a key, which is sharded the (now shutdown) node
  6. connection closed error

an option to fix this would be to provide an option for a slot refresh interval (ioredis does this too), will send a pull request

Node.js Version

No response

Redis Server Version

No response

Node Redis Version

No response

Platform

No response

Logs

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/client/lib/cluster/index.ts, especially the #slots.rediscover() calls shown for MOVED and ASK replies. Reproduce the reshard, del-node, and shutdown sequence, then trace how the outdated cluster topology remains active. Done means the scale-down case refreshes the slots and avoids the reported connection closed error.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, redis, typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.