Redis queue - concurrency issues
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 1.1k
- Forks
- 285
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
I noticed that moveExpired method can cause problems when multiple workers are running simultaneously. If multiple workers call zrevrangebyscore at the same time, before zremrangebyscore is called, both will get the same message IDs and will add them to the waiting queue, causing duplicate entries.
If this happens, one of two problems will occur:
a) Duplicate message ID will be fetched after the job is already processed, causing queue to crash with this exception:
PHP Notice 'yii\base\ErrorException' with message 'Undefined offset: 1'
in vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php:106
This happens because the payload was already deleted and the script doesn't check if it actually got the payload before trying to parse it.
b) Two or more worker processes will get the same duplicated message ID, causing the same job to be executed multiple times in parallel.
Steps to repeat:
- Launch multiple workers (4+ recommended) that listen to the queue with short wait time
- Add a couple thousand jobs to the queue with a delay(30s)
- Wait for errors
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Redis queue implementation and the moveExpired method, then inspect vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php around line 106. Reproduce the race with multiple workers and delayed jobs as described, and verify that concurrent processing no longer creates duplicate IDs or triggers the Undefined offset notice.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php, redis
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100