elsa-workflows / elsa-workflows/elsa-core
[Tracking] Azure Service Bus-backed Elsa startup reliability (orphaned subscriptions, subscription cap, scheduling backlog)
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
## Overview
Tracking issue for a cluster of related problems reported by a customer running Elsa **3.6** on Kubernetes with the **Azure Service Bus (MassTransit) transport**: application startup becomes very slow and Kubernetes crash-loops the pod once Service Bus accumulates entities/backlog. Investigation (with end-to-end and broker-free reproductions) surfaced **multiple distinct root causes** plus one previously-known correlated error. This issue groups them so they can be triaged and fixed coherently.
## Symptom (as reported)
- App startup slow → startup/liveness probe failure → pod killed → entities/backlog grow → next boot worse (compounding).
- Customer workaround: delete topics/queues in Service Bus before redeploying.
## Child issues
- [ ] **#7736 — Random application instance name leaks an ASB change-token subscription/queue on every restart** _(root cause)_
Each process start gets a new random instance name → a new per-instance change-token subscription/queue that is abandoned and never idle-reclaimed, so they accumulate across restarts. **Fixed by PR #7734** (opt-in stable instance name).
- [ ] **#7732 — App startup hangs / K8s crash loop once the change-token topic reaches the 2,000-subscription cap** _(symptom of #7736)_
Once the orphans trip the hard 2,000-subscriptions-per-topic ASB limit, `CreateSubscription` returns 403 `QuotaExceeded`, MassTransit retries forever, and with `WaitUntilStarted=true` the host never reports ready.
- [ ] **#7735 — Startup blocks (and dispatch floods) when many orphaned Delay/Timer/Cron scheduling bookmarks exist** _(distinct, broker-independent)_
`UpdateTenantSchedules` re-schedules every scheduling bookmark synchronously on the host-blocking activation path (linear, no cap); past-due bookmarks additionally fire a 1ms-clamped dispatch storm. Empirically reproduced (0→0.8s, 100k→5.7s).
- [ ] **#7033 — Quartz "Couldn't store trigger … RunWorkflowJob does not exist"** _(pre-existing; present in the customer's logs)_
Same Quartz stacktrace appears in the customer report. Not the cause of slow startup, but part of the same incident.
## Relationships
```
#7736 (random instance name → orphan leak) ──causes──▶ #7732 (cap reached → startup hang)
└─ fixed by PR #7734
#7735 (scheduling bookmark backlog → startup block + dispatch flood) [independent of the broker]
#7033 (RunWorkflowJob trigger error) [pre-existing, co-occurs in customer logs]
```
## Status
- **PR #7734** (opt-in stable instance name) open against `patch/3.6.3` — addresses #7736 and unblocks #7732.
- #7735 filed with code analysis + empirical results; fix directions noted (move re-scheduling off the host-blocking path, throttle past-due catch-up, page queries).
- #7033 already triaged.
## Candidate follow-ups (defense-in-depth, separate PRs)
- Make `MassTransitHostOptions.WaitUntilStarted` configurable in `Elsa.ServiceBus.MassTransit` so a backlog-heavy / topology-stalled boot doesn't gate readiness.
- Lower the default heartbeat/orphan-cleanup timeout so existing `RemoveOrphanedSubscriptions` self-healing reclaims orphans faster.
- Background/throttle `UpdateTenantSchedules`; throttle/jitter past-due timer catch-up; page the trigger/bookmark queries.
- Consider defaulting to stable instance naming in a future minor/major.
Contributor guide
Research direction
Start by reading child issues #7732, #7735, and #7736, then review PR #7734 and the status of the proposed follow-ups. This is a tracking issue rather than a single implementation task; work is done when one independently scoped reliability problem has an agreed fix and validation plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, kubernetes
- Domain
- backend, cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100