Retry logic does not handle unresponsive connections well
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 17.6k
- Forks
- 2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 40
Description
Issue
I was doing experiments of putting Redis behind a toxiproxy service in kubernetes for integration-test automation of network connectivity errors and server-behavior errors and noticed a few things. In this particular setup when the disconnect-mode testing was active, the redis-client was able to tcp-connect to a forwarding proxy, but would get then get disconnected a few milliseconds after it sent data, with no responses. I noticed this caused problems in two ways in the driver:
-
The on_connect() logic would call this.initialize_retry_vars(), causing the attempt count seen in the retryStrategy handler to be perpetually stuck at 1.
-
The use of this.send_offline_queue() in the on_ready() would assume it had a healthy & authenticated connection. At that point, the driver thinks it had run 'auth' and 'select', when actually the 'auth' command had actually failed, and the invocation of this.send_offline_queue() would dequeue and lose all the off-line queued commands on the first failed reconnect.
It seems to me like the code needs two changes to handle this scenario:
-
The on_ready() should validate it has a healthy connection by making sure a 'ping' command is sent and gets a reply before it tries to send the offline queue commands. Perhaps this could be conditioned with a pingOnConnect connect-option.
-
The clearing of attempts & re-initialization in init_retry_vars should only happen after all the off-line queue has been sent.
I ended up working around this issue by changing our test-environment to not provide the faux-connection when the redis was in 'disconnect mode' testing. That said, this seems like something that might occur in the future with various HA load-balancers in front of Redis, so I wanted to get this discussed with the code owners and get their agreement on the solutions above before working on a PR.
Environment
- Node.js Version:
12.16.1
- Redis Version:
redis-10.5.4
-
Platform:
Linux (container on MacOS docker) -
node-redis version: '3.0.2'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the on_connect(), initialize_retry_vars(), on_ready(), send_offline_queue(), and init_retry_vars() entry points mentioned in the report. Reproduce the disconnect-mode scenario with the described toxiproxy setup, then verify that retry attempts are not reset prematurely and queued commands are retained until a healthy authenticated connection is confirmed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, redis, typescript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100