HangfireIO / HangfireIO/Hangfire
A transient storage outage permanently fails a job and orphans its continuations
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
When the storage goes down mid-job, `Worker.TryChangeState` exhausts its 10 attempts and then forces
the `FailedState` with `disableFilters: true` (`Worker.cs:280`; `DelayedJobScheduler.cs:382` does the
same for `Scheduled -> Enqueued`). That flag swaps the filtered pipeline for the bare
`CoreStateMachine`, so:
* `AutomaticRetryAttribute` never converts `Failed` to `Scheduled`/`Deleted` — the job rests in
`Failed` whatever the application's retry policy says (`RetryCount` is never even set);
* `ContinuationsSupportAttribute` never runs, so **every continuation behind that job is orphaned**.
`FailedState.IsFinal` is `false`, so nothing will ever release them.
Permanent, and needs manual intervention. A job that may have *succeeded* is recorded as failed and
takes an unbounded number of continuations down with it.
### Reproduction
Enqueue J1, attach continuations J2..J4 while it is `Processing`, then
`alter database [X] set offline with rollback immediate` and let J1 throw. Bring the database back
once "state change attempts failed" is logged, so the final forced write succeeds:
```
[Error] Worker: 10 state change attempt(s) failed due to an exception, moving job to the FailedState
head job 1: Failed ("Failed to change state to a 'Failed' one ... after 10 retry attempts")
awaiting: 2 -> parent 1 (Failed), 3 -> parent 2, 4 -> parent 3
```
J2..J4 stay `Awaiting` forever.
### Fix
`disableFilters` exists for a *filter* that keeps throwing, as the comment in
`BackgroundJobStateChanger` says — but it currently fires for any exception, including a storage
outage, where the right action is to do nothing and let the invisibility timeout re-deliver the job.
Calling `JobStorage.IsTransientException` (the hook #2589 adds) before the fallback and rethrowing
when it is `true` makes the reproduction above recover on its own. Storages that don't override the
hook keep today's behaviour. I have a patch with tests and can open it once #2589 lands.
Same category error as #2582: a transient infrastructure failure recorded as a permanent job-level
verdict.
The trade-off is re-execution: a job that succeeded but couldn't record it runs again instead of
being marked `Failed`. That falls under the at-least-once contract and beats the current outcome —
but it does put weight on classification. An error that is permanent yet reported as transient would
re-execute the job indefinitely, where today it would at least come to rest. Worth bearing in mind
for #2589's SqlServer override, which treats any `DbException` as transient — broader than the base
method's own documentation advises.
Related: #2582, #2589
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Worker.cs:280, DelayedJobScheduler.cs:382, and the BackgroundJobStateChanger fallback, then inspect JobStorage.IsTransientException and the related #2589 changes. Reproduce the storage outage with chained continuations J1–J4 and observe the forced FailedState path. Done means transient failures are redelivered rather than permanently failing the job, and continuations recover according to the existing retry behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100