HangfireIO / HangfireIO/Hangfire

Job runs/enqueued multiple times if server restarts while job is being processed

Open
#2,380 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

Scenario:
- Configure the server to use sliding visibility
```csharp
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5.0),
QueuePollInterval = TimeSpan.Zero,
```
- Enqueue a job
- wait for it to be in the processing state
- restart the job server while the job is being processed
- The dashboard shows a notification that the server stopped resopnding
- click the re-enqueue button in the dashboard for that job before the job is re-enqueued by the system itself
![screenshot_20240319_132114](https://github.com/HangfireIO/Hangfire/assets/647294/3438fb66-3c5d-4d65-88f4-41def98514da)

Actual result:
- After the sliding visibility timeout is exceeded, a second instance of the job starts.
- That's because the JobQueue table gets another entry for the job and the legacy one is used as well when the FetchedAt timestamp is outrun by the timeout
![screenshot_20240319_131757](https://github.com/HangfireIO/Hangfire/assets/647294/1ccdde3e-03d0-4b4d-ab63-38d13290ff4c)

Expected result:
- A job can not run multiple times concurrently
- There is not more than one entry for each job in the JobQueue table, even if the server restarts/crashes etc.

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported scenario with SlidingInvisibilityTimeout set to five minutes and QueuePollInterval set to zero, then inspect the JobQueue table across a server restart and dashboard re-enqueue. Confirm how the legacy and newly created entries are consumed after FetchedAt expires. Done means a restarted or crashed server cannot leave duplicate JobQueue entries or run the same job concurrently.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.