elsa-workflows / elsa-workflows/elsa-core
Improve JobRunnerHostedService to be more reliably and scalable
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
## Enhancement Request
### Enhancement Overview
When using Elsa I have a problem where blocked tasks cannot be recovered from and seriously degrade the server performance. Investigation into this shows that Elsa basically makes 4 threads to run Tasks. If a Task for an activity throws an exception and allow it to bubble up, then the Task runner ends and it can then only run 3 at a time. 4 exceptions happening in tasks basically kills an Elsa server.
### Proposed Enhancement
I would rather Elsa didn't attempt to limit the number of parallel tasks and instead started them all and allowed the runtime to manage queueing and task creation.
This can be done by changing the `JobRunnerHostedService` to no have a set of 4 worker tasks, but instead just loop through queued tasks starting them and telling them to `ContinueWith` the logging and notification logic.
### Alternative Solutions
An alternative would be to catch all Exceptions from the Tasks and log and then swallow them.
This should prevent exceptions from eventually killing the server, but it will still have the server limited to 4 concurrent Tasks / Activities at a time.
### Use Cases
**Identify potential use cases**
I think this is an improvement to the core scalability and reliability of the platform
### Impact of Enhancement
**Explain the potential impact**
This change will at least prevent unhandled exceptions from taking down a server, but can also improve the level of concurrency in an Elsa server
### Additional Context
**Add any other context**
I did raise this a while ago in the Discord channel and showed my changes to the code.
Contributor guide
Research direction
Start by reading the JobRunnerHostedService implementation and tracing how its four worker tasks consume queued tasks and handle exceptions. Compare the proposed unbounded task-starting and ContinueWith approach with the alternative of catching and logging exceptions. Done means blocked tasks no longer permanently reduce capacity or take down the server, with concurrency and failure behavior covered by the relevant existing tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100