Automattic / Automattic/kue

Redis connection lost and command aborted. code: 'UNCERTAIN_STATE'

Open
#1,091 11 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

We have a node.js app deployed on Heroku and we are using kue to run some background jobs. It usually works fine, but few days back we saw following errors in our production server:

```
Jul 10 11:19:23 refocus app/web.6: Error removing 4628051 { AbortError: Redis connection lost and command aborted. It might have been processed.
Jul 10 11:19:23 refocus app/web.6: at RedisClient.flush_and_error (/app/node_modules/kue/node_modules/redis/index.js:357:23)
Jul 10 11:19:23 refocus app/web.6: at RedisClient.connection_gone (/app/node_modules/kue/node_modules/redis/index.js:659:14)
Jul 10 11:19:23 refocus app/web.6: at Socket. (/app/node_modules/kue/node_modules/redis/index.js:293:14)
Jul 10 11:19:23 refocus app/web.6: at Socket.g (events.js:286:16)
Jul 10 11:19:23 refocus app/web.6: at emitNone (events.js:91:20)
Jul 10 11:19:23 refocus app/web.6: at Socket.emit (events.js:185:7)
Jul 10 11:19:23 refocus app/web.6: at endReadableNT (_stream_readable.js:975:12)
Jul 10 11:19:23 refocus app/web.6: at _combinedTickCallback (internal/process/next_tick.js:74:11)
Jul 10 11:19:23 refocus app/web.6: at process._tickDomainCallback [as _tickCallback] (internal/process/next_tick.js:122:9)
Jul 10 11:19:23 refocus app/web.6: code: 'UNCERTAIN_STATE',
Jul 10 11:19:23 refocus app/web.6: command: 'EXEC',
Jul 10 11:19:23 refocus app/web.6: errors:
Jul 10 11:19:23 refocus app/web.6: [ { AbortError: Redis connection lost and command aborted. It might have been processed.
Jul 10 11:19:23 refocus app/web.6: at RedisClient.flush_and_error (/app/node_modules/kue/node_modules/redis/index.js:357:23)
Jul 10 11:19:23 refocus app/web.6: at RedisClient.connection_gone (/app/node_modules/kue/node_modules/redis/index.js:659:14)
Jul 10 11:19:23 refocus app/web.6: at Socket. (/app/node_modules/kue/node_modules/redis/index.js:293:14)
Jul 10 11:19:23 refocus app/web.6: at Socket.g (events.js:286:16)
Jul 10 11:19:23 refocus app/web.6: at emitNone (events.js:91:20)
Jul 10 11:19:23 refocus app/web.6: at Socket.emit (events.js:185:7)
Jul 10 11:19:23 refocus app/web.6: at endReadableNT (_stream_readable.js:975:12)
Jul 10 11:19:23 refocus app/web.6: at _combinedTickCallback (internal/process/next_tick.js:74:11)
Jul 10 11:19:23 refocus app/web.6: at process._tickDomainCallback [as _tickCallback] (internal/process/next_tick.js:122:9)
Jul 10 11:19:23 refocus app/web.6: code: 'UNCERTAIN_STATE',
Jul 10 11:19:23 refocus app/web.6: command: 'ZREM',
Jul 10 11:19:23 refocus app/web.6: args: [Object],
Jul 10 11:19:23 refocus app/web.6: position: 0 }, .............
```

We have seen these 'UNCERTAIN_STATE' errors before as well. I went through other related issues and it seems a network problem, but since it looks like it is coming from kue, I just wanted to confirm whether we have some potential options to prevent this in future.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the stack-trace locations in node_modules/kue/node_modules/redis/index.js, especially flush_and_error, connection_gone, and the EXEC/ZREM handling. Investigate how Kue handles Redis connection loss and whether the reported UNCERTAIN_STATE condition can be reproduced or documented. Done means establishing a specific preventive or recovery change, with tests or clearly documented operational guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.