Automattic / Automattic/kue

How to processing multiple jobs in a batch?

Open
#930 7 comments 2 reactions 0 assignees View on GitHub
Question
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

Assuming we have many jobs of insert data to a database in the queue, instead of process each jobs one by one, which is not very efficient, I expect to be able to grab multiple jobs, and insert those data in one go by using db's bulk insert command. How can I do that?

Is there anyway to fetch all jobs(or a given number of jobs) from queue, process them and then mark them as done? Thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing job-consumption API and Redis-backed worker behavior. Define how a worker fetches a bounded batch, processes the jobs with one bulk database operation, handles failures, and marks completion. Done means the behavior is documented and covered for successful and partially failing batches.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.