Automattic / Automattic/kue

done callback is not properly completing the job

Open
#1,035 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

I have several jobs which definitely finish execution, and our accounting through PMX custom metrics assure us that the done callbacks are being called and that only 10 concurrent jobs are being run. Despite all of this, we have numerous cases where jobs will remain stuck in the queue as "active".

We do have cleanup routines for these, but with the time sensitivity of some jobs, it's painful to have a message stuck in the queue as active if it's not actually running.

We've added try catch wrappers, and promise catch wrappers at multiple layers to detect some other fault that would somehow call our custom metric's dec() counter but fail to recognize the done() callback.

Is it possible we're raising an exception inside of KUE with too much result data?

```
job_counter.inc();
process_job().then(function (res) {
console.log(new Date(), 'ran job');
job_counter.dec();
try {
done(null, res);
} catch(e) {
console.error('Failed to call done,',e);
done();
}
}).catch(function (err) {
try {
done(err);
} catch (e) {
console.error('Failed to call done,',e);
done();
}
});
```

concurrency is set to 10 and the job_counter is always at 10, yet we get about 1-5% of the jobs hung up in the active state and no "failed to call done" logs.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the worker's done callback path and the provided promise-based job handler, focusing on how jobs transition out of the active state when concurrency is set to 10. Reproduce the reported 1–5% stuck-job behavior and inspect the callback and result-data handling; done means completed jobs no longer remain active and the failure is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.