yiisoft / yiisoft/yii2-queue

Redis queue - concurrency issues

Open
#173 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status:to be verified
Dominant language
PHP
Stars
1.1k
Forks
285
Avg merge
5d 3h
Merged PRs (30d)
2

Description

I noticed that moveExpired method can cause problems when multiple workers are running simultaneously. If multiple workers call zrevrangebyscore at the same time, before zremrangebyscore is called, both will get the same message IDs and will add them to the waiting queue, causing duplicate entries.

If this happens, one of two problems will occur:
a) Duplicate message ID will be fetched after the job is already processed, causing queue to crash with this exception:
PHP Notice 'yii\base\ErrorException' with message 'Undefined offset: 1' in vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php:106
This happens because the payload was already deleted and the script doesn't check if it actually got the payload before trying to parse it.
b) Two or more worker processes will get the same duplicated message ID, causing the same job to be executed multiple times in parallel.

Steps to repeat:

  1. Launch multiple workers (4+ recommended) that listen to the queue with short wait time
  2. Add a couple thousand jobs to the queue with a delay(30s)
  3. Wait for errors

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Redis queue implementation and the moveExpired method, then inspect vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php around line 106. Reproduce the race with multiple workers and delayed jobs as described, and verify that concurrent processing no longer creates duplicate IDs or triggers the Undefined offset notice.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, redis
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.