AB-Law / AB-Law/Pluck-It

`ImageProcessingWorker.cs`: Hard 125s timeout with no jitter — thundering herd on processor slowdown

Open
#61 0 comments 0 reactions 0 assignees View on GitHub
bug performance
Dominant language
TypeScript
Stars
0
Forks
0
Avg merge
16m
Merged PRs (30d)
1

Description

### Problem

All image processing jobs use a fixed 125-second timeout with no jitter:

**File:** `PluckIt.Functions/Functions/ImageProcessingWorker.cs:212`

When the Modal.com segmentation processor is slow (cold start, GPU queue), many jobs hit the timeout simultaneously. They all retry at the same time, causing a thundering herd that overwhelms the processor further.

### Impact

- Correlated retries amplify processor overload during degraded conditions
- Recovery time is longer than necessary — retries pile on rather than spacing out
- Users experience a burst of failures rather than graceful degradation

### Proposed Fix

Add exponential backoff with jitter on retry:

```csharp
var delay = TimeSpan.FromSeconds(Math.Pow(2, attempt) + Random.Shared.NextDouble() * 5);
await Task.Delay(delay, cancellationToken);
```

Consider also reducing the initial timeout (e.g. 60s) and increasing max retry count, rather than one long synchronous wait.

### Functionality Impact

Slower worst-case recovery for a single job, but significantly better system-wide behaviour under load.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.