payloadcms / payloadcms/payload

Add jitter to exponential backoff in job queue retry

Open Beginner friendly
#16,318 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
TypeScript
Stars
44.8k
Forks
4.2k
Avg merge
2d 21h
Merged PRs (30d)
53

Description

Summary

The exponential backoff calculation in packages/payload/src/queues/errors/calculateBackoffWaitUntil.ts (line 24) uses a deterministic delay formula:

waitUntil = new Date(now.getTime() + Math.pow(2, totalTried) * delay)

This means concurrent jobs that fail at the same time will all retry at exactly the same intervals (2^n * delay), creating a thundering herd effect on the queue backend.

Suggested fix

Add decorrelated jitter to spread retry times:

const baseDelay = Math.pow(2, totalTried) * delay
const jitter = baseDelay * (0.5 + Math.random() * 0.5)
waitUntil = new Date(now.getTime() + jitter)

This preserves the exponential growth curve while adding ±50% randomization, which is standard practice for distributed retry (AWS, Google Cloud, and most resilience libraries recommend this).

Why this matters

In production job queues with many workers, deterministic backoff causes retry storms — all failed jobs retry simultaneously, overloading the backend at predictable intervals. Jitter breaks the synchronization and spreads load.

Impact

  • Additive change, no API modification
  • Existing behavior preserved within ±50% timing variance
  • No new dependencies
  • The fixed backoff type is unaffected (only exponential changes)

Happy to submit a PR if this approach looks right.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/payload/src/queues/errors/calculateBackoffWaitUntil.ts at line 24 and inspect how exponential and fixed backoff types are handled. Update the exponential delay to include the proposed jitter while leaving fixed backoff unchanged, then verify that retry times are randomized around the exponential delay.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.