elsa-workflows / elsa-workflows/elsa-core

Bad performances when using lots of scheduled Hangfire Jobs

Open
#4,565 1 comment 0 reactions 0 assignees View on GitHub
triaged
Dominant language
C#
Stars
7.9k
Forks
1.5k
Avg merge
15h 22m
Merged PRs (30d)
114

Description

Hello,
we are having performance issues in out ELSA 2.13.0 instance, up to the point where a single workflow burst takes from 6 to 10 seconds.
I traced the problem down to its root, but I need some advice on how to fix it before making a pull request.

Here is the issue: the ELSA instance contains currently 20k workflow instances, and most of them use the "StartAt" activity, using Hangfire as a backend. Both ELSA and Hangfire use SQL Server as their persistence backend.
When I started loading the system with instances, I noticed a progressive performance degrade. After some debugging, the most significant amount of time seems to be the `Elsa.Activities.Temporal.Hangfire.Extensions.JobStorageExtensions.EnumerateScheduledJobs` extension method contained in the `Elsa.Activities.Temporal.Hangfire` project.

As far as I understand, this method is called by the `HangfireWorkflowDefinitionScheduler` and `HangfireWorkflowInstanceScheduler` every time a suspended activity is executed, through the `UnscheduleTimers` handled class.

The `EnumerateScheduledJobs` is slow because the interface exposed by the Hangfire `IMonitoringApi` class only allows to load **all** scheduled jobs, without any means of filtering it on the persistence side; this results in the in-memory linear search of all scheduled jobs by every activity execution in the system.

Currently, I hacked a very dirty way of solving the problem and keep our project going, but I would like to bring a more structured (and better performing) solution.

Here is what I would like to do:

- Whenever a job is scheduled, Hangfire returns its unique id, so we can associate it to the workflowInstanceId, workflowDefinitionId, activityId that generated it;
- Add a persisted Entity (and here is the main point where I need help) only used (and installed?) by the `Elsa.Activities.Temporal.Hangfire` module, keeping track of the association.
- When we need to cleanup the scheduled jobs, we query the association directly, and use Hangfire only to actually unschedule them, just like now.

I also thought about saving the Job Id in the activity/workflow but the fact that the temporal activities are abstract w.r.t. the underlying engine makes it unpractical in my opition.

So, do you think you are willing to integrate a change like this? If so, could you please point me to what I need to be able to add a persisted entity?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.