redis / redis/node-redis

Retry logic does not handle unresponsive connections well

Open
#1,510 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
17.6k
Forks
2k
Avg merge
2d 3h
Merged PRs (30d)
40

Description

Issue

I was doing experiments of putting Redis behind a toxiproxy service in kubernetes for integration-test automation of network connectivity errors and server-behavior errors and noticed a few things. In this particular setup when the disconnect-mode testing was active, the redis-client was able to tcp-connect to a forwarding proxy, but would get then get disconnected a few milliseconds after it sent data, with no responses. I noticed this caused problems in two ways in the driver:

  1. The on_connect() logic would call this.initialize_retry_vars(), causing the attempt count seen in the retryStrategy handler to be perpetually stuck at 1.

  2. The use of this.send_offline_queue() in the on_ready() would assume it had a healthy & authenticated connection. At that point, the driver thinks it had run 'auth' and 'select', when actually the 'auth' command had actually failed, and the invocation of this.send_offline_queue() would dequeue and lose all the off-line queued commands on the first failed reconnect.

It seems to me like the code needs two changes to handle this scenario:

  1. The on_ready() should validate it has a healthy connection by making sure a 'ping' command is sent and gets a reply before it tries to send the offline queue commands. Perhaps this could be conditioned with a pingOnConnect connect-option.

  2. The clearing of attempts & re-initialization in init_retry_vars should only happen after all the off-line queue has been sent.

I ended up working around this issue by changing our test-environment to not provide the faux-connection when the redis was in 'disconnect mode' testing. That said, this seems like something that might occur in the future with various HA load-balancers in front of Redis, so I wanted to get this discussed with the code owners and get their agreement on the solutions above before working on a PR.


Environment
  • Node.js Version: 12.16.1
  • Redis Version: redis-10.5.4
  • Platform: Linux (container on MacOS docker)

  • node-redis version: '3.0.2'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the on_connect(), initialize_retry_vars(), on_ready(), send_offline_queue(), and init_retry_vars() entry points mentioned in the report. Reproduce the disconnect-mode scenario with the described toxiproxy setup, then verify that retry attempts are not reset prematurely and queued commands are retained until a healthy authenticated connection is confirmed.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, redis, typescript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.