Automattic / Automattic/kue

How do I "do something" only at the end of all jobs when using Processing Concurrency?

Open
#525 9 comments 0 reactions 0 assignees View on GitHub
Discussion Question
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

I have a queue that is going to be handling **massive** amounts of data. I have set the processing concurrency to `2000`. What I want to do is iterate over all jobs in the _batch_, then when I have iterated over all I want to save to a database. Something like this pseudocode:
- Incoming batch of 2000 jobs
- Before iteration, initialize some arrays that will hold aggregated data
- Aggregate data from 2000 jobs into arrays
- Save aggregated arrays into DB

Any ideas?

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue mentions Processing Concurrency, batches, and a queue backed by Redis, but names no files, tests, or entry points. Start by reviewing Kue's batch and concurrency behavior and the nine existing comments; done would require a clear, supported way to run aggregation after all jobs complete, or documentation explaining that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.