Automattic / Automattic/kue

Jobs Getting Stuck in Active State

Open
#514 38 comments 0 reactions 0 assignees View on GitHub
Duplicate Question
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

Hi @behrad I am still experiencing jobs being stuck in the `active` state, similar to issue #391

I am using `kue@0.8.11` and redis `2.8.17`

I have tried to carefully setup all recommended steps to avoid this. Here are some code snippets. If I'm doing something wrong I would love to hear about it. I've just torn out the bits of source that deal with kue/jobs here and redacted stuff that wasn't relevant.

_producer_ -- separate node process

``` javascript
var jobs = kue.createQueue({ disableSearch: true, redis: config.redis });

var dying = false;
process.on('uncaughtException', function (err) {
log.error('uncaught exception', err.stack);
die();
dying = true;
});

process.on('SIGTERM', function () {
log.error('SIGTERM');
die();
dying = true;
});

function die() {
if (!dying) {
jobs.shutdown(function (err) {
if (err) { log.error('Kue DID NOT shutdown gracefully', err); }
else { log.info('Kue DID shutdown gracefully'); }
process.exit(1);
}
}

// createJob( ) is called on an interval as needed

function createJob(data) {
var job = jobs.create('deliver:sub', { title: 'Deliver Subscription', sub: data })
.removeOnComplete(true)
.attempts(5)
.backoff({type: 'exponential'})
.save(function (err) {
if (err) { return log.error('Error creating deliver:sub', err); }
log.info(util.format('job[%s] created', job.id));
});

job.on('complete', function() {
/* logic here */
});

job.on('failed', function () {
log.error(util.format('job[%s] failed ', job.id));
});
}
```

_consumer_ -- separate node process

``` javascript
var jobs = kue.createQueue({ disableSearch: true, redis: config.redis });

try {
kue.app.listen(3001);
} catch (err) {
log.error('Error: could not start kue express server', err);
}

// called in master b/c we use job retries
jobs.promote();

jobs.watchStuckJobs();

var dying = false;
process.on('uncaughtException', function (err) {
log.error('uncaught exception', err.stack);
die();
dying = true;
});

process.on('SIGTERM', function () {
log.error('SIGTERM');
die();
dying = true;
});

function die() {
if (!dying) {
jobs.shutdown(function (err) {
if (err) { log.error('Kue DID NOT shutdown gracefully ', err); }
else { log.info('Kue DID shutdown gracefully'); }
process.exit(1);
});
}
}

jobs.process('deliver:sub', 5, function (job, done) {
var domain = require('domain').create();

domain.on('error', function (err) {
log.error('domain error for deliver:sub', err);
done(err);
});

domain.run(function () {
process.nextTick(function () {
/* logic here that ends with either */
done();
/* OR */
done(err);
});
});
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the issue's use of jobs.process(), jobs.watchStuckJobs(), jobs.promote(), and jobs.shutdown() with Kue 0.8.11 and Redis 2.8.17, then compare the behavior with issue #391. Reproduce the active-state condition before narrowing down the relevant queue lifecycle path; done means identifying a reproducible cause and documenting a verified resolution or regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.