Performance problem under high load
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
Hi, firstly thanks for this cool library.
If the number of jobs being created is not too much, this library is perfect. I wanted to use this library in a heavy load scenario. Unfortunately results of my experiments were not so bright.
I'm working on > 100.000 keys of sample data. In Redis:
```
debug populate 100000 my:namespace
```
Here is my job creator:
``` javascript
var kue = require('kue');
var ioredis = require('ioredis');
var redis = ioredis();
var jobs = kue.createQueue();
var stream = redis.scanStream({
match: 'my:namespace:*',
count: 100
});
var keys = [];
stream.on('data', function(resultKeys) {
resultKeys.forEach(function(item, i) {
console.log(i, item);
keys.push(item);
})
});
stream.on('end', function() {
console.log("FINISHED!");
keys.forEach(function(item) {
jobs.create('ad_fetch', {
item: item
}).save();
})
});
jobs.on('error', function(err) {
console.log("There was an error!:");
console.log(err);
})
```
And here is my worker code:
``` javascript
var kue = require('kue');
var jobs = kue.createQueue();
jobs.process('ad_fetch', 100, function(job, done) {
var data = job.data.item
console.log(data + " is BEING PROCESSED!");
done();
});
```
In this scenario, it takes 2 seconds to fetch 100.000 records from Redis using `.scan` stream. After the streaming ends and I have all keys inside an array and I iterate over it to create jobs for per key returned from Redis. Even though stream is complete, it takes almost 13 seconds for jobs to be created. I know that it's the time elapsed during necessary key/value creation in Redis. But, frankly, I think there should be some stuff to be optimized. For example why does the queue should wait for all jobs to be saved to start executing?
Am I doing something wrong? If there is a better way to do achieve what I'm trying to do, I'd be glad if you share it. Thank you!
Update: By the way, I thought maybe the reason could be limitations of a single Redis instance and I installed a second Redis instance. I seperated them in my sample app. I mean data is received from Redis1 and jobs data is saved to Redis2... Nothing changed. Same performance.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue with the provided Redis SCAN stream, job-creator script, and worker using 100,000 keys. Measure the time spent fetching keys, creating jobs, and processing them, including the separate-Redis setup described in the update. Done means identifying an actionable bottleneck or confirming that the behavior requires a design decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100