Automattic / Automattic/kue

KUE 0.9.4 not working properly with redis-sentinel (latest)

Open
#739 3 comments 0 reactions 0 assignees View on GitHub
need more info
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

Hi Team,
Following is the scenerio in our NODEJS application.
we are using REDIS1 (Master) & REDIS2 (Slave) via sentinel to connect to redis and do automatic failover when the master stops working. Everyother connection taken manually from the redis-sentinel library works fine and it connects automatically if the conn drop to the master (which goes down). this all happens perfectly as stated in the docs.
However, the problem arises when we are trying to use REDIS-SENTINEL with KUE. Once KUE establish the connection with one of the master via Sentinel PUB/SUB works fine. as soon as the Master is being changed by the Sentinel the connection maintained by the KUE (as per the logs) seems to be perfectly reconnected to the new master. However, another client which publishes something on a channel (to which the listener is listening in this application) try to PUBLISH anything, the KUE doen't get the notification. i..e the Worker created via QUEUE does not listen anything. and all the messages sent to the channel appeared in the KUE UI as queued.
We followed the same set of examples as being provided on the site.
Here is the KUE configuration.

var redisEndpoints = [
{host: '192.168.191.1', port: 26379},
{host: '192.168.191.2', port: 26379}
];
var opts = {}; // Standard node_redis client options , like Passwords etc.
var masterName = 'mymaster';
var sentinel = Sentinel.Sentinel(redisEndpoints);

var que = kue.createQueue({
redis: {
createClientFactory: function(){
//return sentinel.createClient(redisEndpoints, masterName, opts);
return sentinel.createClient(masterName, opts);
}
}
});
que.watchStuckJobs();

we are stuck because of this. Any help on this will be appreciated

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Kue configuration using createClientFactory and sentinel.createClient, then reproduce a Redis Sentinel master failover while a worker listens for queue notifications. Verify whether the worker receives newly published jobs after failover and use the Kue logs and Redis Sentinel behavior to narrow down the reconnect path.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, redis
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.