cakephp / cakephp/queue

Issue with multiple consumers - re-delivering in progress jobs

Open
#110 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
PHP
Stars
36
Forks
22
Avg merge
20h 36m
Merged PRs (30d)
2

Description

Hello,

I'm not sure if I have found a bug or if this just a case of user error.

I am having an issue with running multiple workers on long running jobs (~20 minutes).

When add a message into the queue and then start 2 workers using the following command
bin/cake worker -c process-invoices -p process-invoices

Initially, 1 worker will begin to process the job. However, after about 5 minutes, the other worker also begins to start working on the same job.

I have set $maxAttempts = 1 in the job class.

Next, I tried to bind each processor to an unique name
bin/cake worker -c process-invoices -p process-invoices1
and
bin/cake worker -c process-invoices -p process-invoices2

The message is only picked up by 1 worker, however if another message gets added to the queue, the other worker receives the following error:

[LogicException] Processor was not found. processorName: "process-invoices2" in /var/www/html/vendor/enqueue/enqueue/ArrayProcessorRegistry.php on line 33

And when I don't bind the processor name and run the following command twice to create 2 workers
bin/cake worker -c process-invoices

I receive a similar error:

error: [LogicException] Processor was not found. processorName: "Cake\Queue\Queue\Processor63e2a1884f1c4" in /var/www/html/vendor/enqueue/enqueue/ArrayProcessorRegistry.php on line 33

For reference, the config for the queue is:

'Queue' => [
    'process-invoices' => [
        'url' => null,
        'queue' => 'process-invoices',
        'logger' => 'stdout',
        'receiveTimeout' => 10000,
        'storeFailedJobs' => true,
    ],
],

I am unable to figure out a method of having multiple workers without the errors or re-processing jobs.
Is there any config setting (visibility timeout) that I am missing to resolve my issues?

Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with two bin/cake worker processes using the process-invoices configuration and a long-running job. Read the worker entry point and the reported ArrayProcessorRegistry.php failure, then compare the behavior with unique and shared processor names. Done means multiple workers can process queued jobs without re-delivery or Processor was not found errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.