How do I "do something" only at the end of all jobs when using Processing Concurrency?
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
I have a queue that is going to be handling **massive** amounts of data. I have set the processing concurrency to `2000`. What I want to do is iterate over all jobs in the _batch_, then when I have iterated over all I want to save to a database. Something like this pseudocode:
- Incoming batch of 2000 jobs
- Before iteration, initialize some arrays that will hold aggregated data
- Aggregate data from 2000 jobs into arrays
- Save aggregated arrays into DB
Any ideas?
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue mentions Processing Concurrency, batches, and a queue backed by Redis, but names no files, tests, or entry points. Start by reviewing Kue's batch and concurrency behavior and the nine existing comments; done would require a clear, supported way to run aggregation after all jobs complete, or documentation explaining that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100