Automattic / Automattic/kue

Performance problem under high load

Open
#765 3 comments 0 reactions 0 assignees View on GitHub
need more info unknown
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

Hi, firstly thanks for this cool library.

If the number of jobs being created is not too much, this library is perfect. I wanted to use this library in a heavy load scenario. Unfortunately results of my experiments were not so bright.

I'm working on > 100.000 keys of sample data. In Redis:

```
debug populate 100000 my:namespace
```

Here is my job creator:

``` javascript
var kue = require('kue');
var ioredis = require('ioredis');
var redis = ioredis();
var jobs = kue.createQueue();

var stream = redis.scanStream({
match: 'my:namespace:*',
count: 100
});

var keys = [];
stream.on('data', function(resultKeys) {
resultKeys.forEach(function(item, i) {
console.log(i, item);
keys.push(item);
})

});
stream.on('end', function() {
console.log("FINISHED!");
keys.forEach(function(item) {
jobs.create('ad_fetch', {
item: item
}).save();
})
});

jobs.on('error', function(err) {
console.log("There was an error!:");
console.log(err);
})
```

And here is my worker code:

``` javascript
var kue = require('kue');
var jobs = kue.createQueue();

jobs.process('ad_fetch', 100, function(job, done) {
var data = job.data.item
console.log(data + " is BEING PROCESSED!");
done();

});
```

In this scenario, it takes 2 seconds to fetch 100.000 records from Redis using `.scan` stream. After the streaming ends and I have all keys inside an array and I iterate over it to create jobs for per key returned from Redis. Even though stream is complete, it takes almost 13 seconds for jobs to be created. I know that it's the time elapsed during necessary key/value creation in Redis. But, frankly, I think there should be some stuff to be optimized. For example why does the queue should wait for all jobs to be saved to start executing?

Am I doing something wrong? If there is a better way to do achieve what I'm trying to do, I'd be glad if you share it. Thank you!

Update: By the way, I thought maybe the reason could be limitations of a single Redis instance and I installed a second Redis instance. I seperated them in my sample app. I mean data is received from Redis1 and jobs data is saved to Redis2... Nothing changed. Same performance.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with the provided Redis SCAN stream, job-creator script, and worker using 100,000 keys. Measure the time spent fetching keys, creating jobs, and processing them, including the separate-Redis setup described in the update. Done means identifying an actionable bottleneck or confirming that the behavior requires a design decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.