elsa-workflows / elsa-workflows/elsa-core

[Tracking] Azure Service Bus-backed Elsa startup reliability (orphaned subscriptions, subscription cap, scheduling backlog)

Open
#7,737 2 comments 0 reactions 0 assignees View on GitHub
bug performance triaged
Dominant language
C#
Stars
7.9k
Forks
1.5k
Avg merge
15h 22m
Merged PRs (30d)
114

Description

## Overview

Tracking issue for a cluster of related problems reported by a customer running Elsa **3.6** on Kubernetes with the **Azure Service Bus (MassTransit) transport**: application startup becomes very slow and Kubernetes crash-loops the pod once Service Bus accumulates entities/backlog. Investigation (with end-to-end and broker-free reproductions) surfaced **multiple distinct root causes** plus one previously-known correlated error. This issue groups them so they can be triaged and fixed coherently.

## Symptom (as reported)

- App startup slow → startup/liveness probe failure → pod killed → entities/backlog grow → next boot worse (compounding).
- Customer workaround: delete topics/queues in Service Bus before redeploying.

## Child issues

- [ ] **#7736 — Random application instance name leaks an ASB change-token subscription/queue on every restart** _(root cause)_
Each process start gets a new random instance name → a new per-instance change-token subscription/queue that is abandoned and never idle-reclaimed, so they accumulate across restarts. **Fixed by PR #7734** (opt-in stable instance name).
- [ ] **#7732 — App startup hangs / K8s crash loop once the change-token topic reaches the 2,000-subscription cap** _(symptom of #7736)_
Once the orphans trip the hard 2,000-subscriptions-per-topic ASB limit, `CreateSubscription` returns 403 `QuotaExceeded`, MassTransit retries forever, and with `WaitUntilStarted=true` the host never reports ready.
- [ ] **#7735 — Startup blocks (and dispatch floods) when many orphaned Delay/Timer/Cron scheduling bookmarks exist** _(distinct, broker-independent)_
`UpdateTenantSchedules` re-schedules every scheduling bookmark synchronously on the host-blocking activation path (linear, no cap); past-due bookmarks additionally fire a 1ms-clamped dispatch storm. Empirically reproduced (0→0.8s, 100k→5.7s).
- [ ] **#7033 — Quartz "Couldn't store trigger … RunWorkflowJob does not exist"** _(pre-existing; present in the customer's logs)_
Same Quartz stacktrace appears in the customer report. Not the cause of slow startup, but part of the same incident.

## Relationships

```
#7736 (random instance name → orphan leak) ──causes──▶ #7732 (cap reached → startup hang)
└─ fixed by PR #7734
#7735 (scheduling bookmark backlog → startup block + dispatch flood) [independent of the broker]
#7033 (RunWorkflowJob trigger error) [pre-existing, co-occurs in customer logs]
```

## Status

- **PR #7734** (opt-in stable instance name) open against `patch/3.6.3` — addresses #7736 and unblocks #7732.
- #7735 filed with code analysis + empirical results; fix directions noted (move re-scheduling off the host-blocking path, throttle past-due catch-up, page queries).
- #7033 already triaged.

## Candidate follow-ups (defense-in-depth, separate PRs)

- Make `MassTransitHostOptions.WaitUntilStarted` configurable in `Elsa.ServiceBus.MassTransit` so a backlog-heavy / topology-stalled boot doesn't gate readiness.
- Lower the default heartbeat/orphan-cleanup timeout so existing `RemoveOrphanedSubscriptions` self-healing reclaims orphans faster.
- Background/throttle `UpdateTenantSchedules`; throttle/jitter past-due timer catch-up; page the trigger/bookmark queries.
- Consider defaulting to stable instance naming in a future minor/major.

Contributor guide

Open the contributing guide

Research direction

Start by reading child issues #7732, #7735, and #7736, then review PR #7734 and the status of the proposed follow-ups. This is a tracking issue rather than a single implementation task; work is done when one independently scoped reliability problem has an agreed fix and validation plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, kubernetes
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.