elsa-workflows / elsa-workflows/elsa-core

Improve JobRunnerHostedService to be more reliably and scalable

Open
#6,365 0 comments 1 reaction 0 assignees View on GitHub
triaged
Dominant language
C#
Stars
7.9k
Forks
1.5k
Avg merge
15h 22m
Merged PRs (30d)
114

Description

## Enhancement Request

### Enhancement Overview
When using Elsa I have a problem where blocked tasks cannot be recovered from and seriously degrade the server performance. Investigation into this shows that Elsa basically makes 4 threads to run Tasks. If a Task for an activity throws an exception and allow it to bubble up, then the Task runner ends and it can then only run 3 at a time. 4 exceptions happening in tasks basically kills an Elsa server.

### Proposed Enhancement
I would rather Elsa didn't attempt to limit the number of parallel tasks and instead started them all and allowed the runtime to manage queueing and task creation.
This can be done by changing the `JobRunnerHostedService` to no have a set of 4 worker tasks, but instead just loop through queued tasks starting them and telling them to `ContinueWith` the logging and notification logic.

### Alternative Solutions
An alternative would be to catch all Exceptions from the Tasks and log and then swallow them.
This should prevent exceptions from eventually killing the server, but it will still have the server limited to 4 concurrent Tasks / Activities at a time.

### Use Cases
**Identify potential use cases**
I think this is an improvement to the core scalability and reliability of the platform

### Impact of Enhancement
**Explain the potential impact**
This change will at least prevent unhandled exceptions from taking down a server, but can also improve the level of concurrency in an Elsa server

### Additional Context
**Add any other context**
I did raise this a while ago in the Discord channel and showed my changes to the code.

Contributor guide

Open the contributing guide

Research direction

Start by reading the JobRunnerHostedService implementation and tracing how its four worker tasks consume queued tasks and handle exceptions. Compare the proposed unbounded task-starting and ContinueWith approach with the alternative of catching and logging exceptions. Done means blocked tasks no longer permanently reduce capacity or take down the server, with concurrency and failure behavior covered by the relevant existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.