elsa-workflows / elsa-workflows/elsa-core
Bad performances when using lots of scheduled Hangfire Jobs
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
Hello,
we are having performance issues in out ELSA 2.13.0 instance, up to the point where a single workflow burst takes from 6 to 10 seconds.
I traced the problem down to its root, but I need some advice on how to fix it before making a pull request.
Here is the issue: the ELSA instance contains currently 20k workflow instances, and most of them use the "StartAt" activity, using Hangfire as a backend. Both ELSA and Hangfire use SQL Server as their persistence backend.
When I started loading the system with instances, I noticed a progressive performance degrade. After some debugging, the most significant amount of time seems to be the `Elsa.Activities.Temporal.Hangfire.Extensions.JobStorageExtensions.EnumerateScheduledJobs` extension method contained in the `Elsa.Activities.Temporal.Hangfire` project.
As far as I understand, this method is called by the `HangfireWorkflowDefinitionScheduler` and `HangfireWorkflowInstanceScheduler` every time a suspended activity is executed, through the `UnscheduleTimers` handled class.
The `EnumerateScheduledJobs` is slow because the interface exposed by the Hangfire `IMonitoringApi` class only allows to load **all** scheduled jobs, without any means of filtering it on the persistence side; this results in the in-memory linear search of all scheduled jobs by every activity execution in the system.
Currently, I hacked a very dirty way of solving the problem and keep our project going, but I would like to bring a more structured (and better performing) solution.
Here is what I would like to do:
- Whenever a job is scheduled, Hangfire returns its unique id, so we can associate it to the workflowInstanceId, workflowDefinitionId, activityId that generated it;
- Add a persisted Entity (and here is the main point where I need help) only used (and installed?) by the `Elsa.Activities.Temporal.Hangfire` module, keeping track of the association.
- When we need to cleanup the scheduled jobs, we query the association directly, and use Hangfire only to actually unschedule them, just like now.
I also thought about saving the Job Id in the activity/workflow but the fact that the temporal activities are abstract w.r.t. the underlying engine makes it unpractical in my opition.
So, do you think you are willing to integrate a change like this? If so, could you please point me to what I need to be able to add a persisted entity?
Contributor guide
Assessment
This issue has not been assessed yet.