payloadcms / payloadcms/payload
Add jitter to exponential backoff in job queue retry
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 44.8k
- Forks
- 4.2k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 53
Description
Summary
The exponential backoff calculation in packages/payload/src/queues/errors/calculateBackoffWaitUntil.ts (line 24) uses a deterministic delay formula:
waitUntil = new Date(now.getTime() + Math.pow(2, totalTried) * delay)
This means concurrent jobs that fail at the same time will all retry at exactly the same intervals (2^n * delay), creating a thundering herd effect on the queue backend.
Suggested fix
Add decorrelated jitter to spread retry times:
const baseDelay = Math.pow(2, totalTried) * delay
const jitter = baseDelay * (0.5 + Math.random() * 0.5)
waitUntil = new Date(now.getTime() + jitter)
This preserves the exponential growth curve while adding ±50% randomization, which is standard practice for distributed retry (AWS, Google Cloud, and most resilience libraries recommend this).
Why this matters
In production job queues with many workers, deterministic backoff causes retry storms — all failed jobs retry simultaneously, overloading the backend at predictable intervals. Jitter breaks the synchronization and spreads load.
Impact
- Additive change, no API modification
- Existing behavior preserved within ±50% timing variance
- No new dependencies
- The
fixedbackoff type is unaffected (onlyexponentialchanges)
Happy to submit a PR if this approach looks right.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/payload/src/queues/errors/calculateBackoffWaitUntil.ts at line 24 and inspect how exponential and fixed backoff types are handled. Update the exponential delay to include the proposed jitter while leaving fixed backoff unchanged, then verify that retry times are randomized around the exponential delay.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100