redis / redis/node-redis

Connection in unrecoverable state

Open
#1,114 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
17.6k
Forks
2k
Avg merge
2d 3h
Merged PRs (30d)
40

Description

There might be an issue with the auto-reconnect feature in redis client. It has been present since at least version 2.4.2 and has been confirmed up to version 2.6.

  • Description: We use Kue for messaging in node and internally it uses Redis as the broker. In our case, it creates a pub/sub client (two in fact, one for sub another for pub). The problem arises when we have too much load on our server and the client output buffer limit is reached (I think this is partially our fault as sometimes the message data might be quite big).
    Checking the redis logs I see cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits. And on the client side I see the 'end' event and the reconnection attempt. The publish client is connected just fine but on the subscribe client I get a 'connect' event, but not a 'ready' event. When things are operating normally I always get a 'ready' soon there after. In this case, no 'ready' and no 'error' is thrown anywhere so the connection appears alive but in fact stops receiving any data from Redis and we have to restart our server.
  • Version: node-redis v2.4.2 to v2.6.0 confirmed. Redis v3.0.7.
  • Platform: Node.js 0.12 on Ubuntu 14.04
  • Steps to reproduce: In order to reproduce this error in our dev environment we had to set the client output buffer limits to something insignificant. In our case it was:
    config set "client-output-buffer-limit" "normal 0 0 0 slave 268435456 67108864 60 pubsub 2 1 1". Then open your pub/sub connection and publish a lot of data in a very short interval (in our case was a few tens of requests with a bit of data in them).
  • Workaround: From the client side we were able to detect this issue by using a timeout function in the 'connect' event if no 'ready' events were received in a sensible time. Then we simply would manually re-create our clients (could not find a way to gracefully instruct redis client to reconnect from the 'connect' event handler).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the auto-reconnect flow around the client's connect and ready events, then reproduce the failure using Redis's client-output-buffer-limit setting and a pub/sub connection. Done means a client closed for exceeding the output buffer reconnects into a usable state, emits the expected readiness signal, and resumes receiving data without manual client recreation.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.