[Tracking] Scheduling startup backlog / past-due catch-up (partially mitigated by #7746/#7747; residual #8155/#8156)

## Summary

Separate from the Azure Service Bus subscription-cap hang in #7732, code analysis of Elsa 3.6 found a **second, distinct startup-blocking path** in the **scheduling** subsystem that does not depend on any transport or hard cap. It degrades startup **gradually** with the number of orphaned `Delay`/`Timer`/`Cron` scheduling bookmarks in the database, which matches reports of slow startup / Kubernetes crash-loops occurring **without** having reached the 2,000-subscription ASB limit.

There are two compounding effects: (1) a synchronous, host-blocking re-scheduling loop on the startup path, and (2) an immediate-fire dispatch flood for past-due bookmarks.

## (1) Host-blocking re-scheduling loop

`ActivateTenants` is an `IHostedService` whose `StartAsync` the .NET Generic Host **awaits before the app reports ready**:

- `src/modules/Elsa.Common/Multitenancy/HostedServices/ActivateTenants.cs` → `tenantService.ActivateTenantsAsync(...)`

Tenant activation awaits the `ITenantActivatedEvent` handlers, including:

- `src/modules/Elsa.Scheduling/Handlers/UpdateTenantSchedules.cs` — `TenantActivatedAsync`:
```csharp
var triggers = (await GetTriggersAsync(...)).ToList(); // triggerStore.FindManyAsync(ByNames([Cron,Timer,Delay]))
var bookmarks = (await GetBookmarksAsync(...)).ToList(); // bookmarkStore.FindManyAsync(ByActivityTypeNames([Cron,Timer,Delay]))
await triggerScheduler.ScheduleAsync(triggers, ...);
await bookmarkScheduler.ScheduleAsync(bookmarks, ...);
```

This loads **all** Cron/Timer/Delay triggers and bookmarks into memory and then:

- `src/modules/Elsa.Scheduling/Services/DefaultBookmarkScheduler.cs` — `ScheduleAsync` iterates **every** bookmark and `await`s `IWorkflowScheduler.ScheduleAtAsync(...)` **sequentially** (O(N) on the blocking startup path).

## (2) Immediate-fire dispatch flood for past-due bookmarks

`DefaultBookmarkScheduler → DefaultWorkflowScheduler.ScheduleAtAsync → LocalScheduler.ScheduleAsync → SpecificInstantSchedule → ScheduledSpecificInstantTask`:

- `src/modules/Elsa.Scheduling/ScheduledTasks/ScheduledSpecificInstantTask.cs` — `Schedule()`:
```csharp
var delay = _startAt - now;
if (delay <= TimeSpan.Zero)
{
_logger.LogWarning("Calculated delay is {Delay} which is not positive. Using minimum delay of 1ms ...");
delay = TimeSpan.FromMilliseconds(1);
}
// timer.Elapsed => commandSender.SendAsync(new RunScheduledTask(_task)) => resume/dispatch the workflow instance
```

Every **past-due** bookmark (`ResumeAt`/`ExecuteAt` in the past — typical for instances suspended on `Delay`/`Timer` that never resumed before a restart) fires within ~1ms. With thousands of them, thousands of `RunScheduledTask` → `ResumeWorkflowTask` → workflow executions hit the mediator, persistence store, and thread pool **simultaneously at boot**. Even though execution runs off the `StartAsync` thread, the saturation starves the remaining startup hosted services and the readiness endpoint → probe timeout → pod is killed.

## Why this matches the field reports

- **No hard cap required** — degradation is proportional to the count of orphaned scheduling bookmarks, so slowdown appears well before any ASB limit.
- Orphaned `Delay`/`Timer` bookmarks accumulate from the same churn (workflows suspended on a timer that never resume due to crashes/redeploys) that accumulates orphaned ASB subscriptions in #7732 — so both issues co-occur in the same deployments.
- ⚠️ Deleting Service Bus queues/topics does **not** remove these DB bookmarks. If a customer reports relief purely from deleting queues, this path may be secondary for them; if relief also involved DB pruning or instances finally draining, this path is implicated. Worth confirming.

## Other startup-blocking tasks observed (lower risk)

- `src/modules/Elsa.Http/Tasks/UpdateRouteTableStartupTask.cs` — loads all HttpEndpoint triggers + bookmarks and builds routes at boot (O(K)).
- `src/modules/Elsa.Workflows.Runtime/Tasks/PopulateRegistriesStartupTask.cs` — scans/deserializes all workflow definitions, indexes triggers, fans out notifications (O(defs); scales with definitions, not instance backlog).

## Suggested reproduction (DB-driven, cap-free)

1. EFCore persistence. A trivial workflow: `Delay` → no-op (cheap to resume).
2. Seed N (e.g. 5k–50k) suspended instances + `StoredBookmark` rows for the `Delay` activity type with `ResumeAt` in the **past** (start N instances and let them suspend, then back-date `ResumeAt`; or bulk-insert bookmark rows with a crafted `DelayPayload`).
3. Restart and measure time from process start to `ApplicationStarted`/readiness vs an empty-DB baseline; vary N to show gradual degradation.
4. Future-dated variant (`ResumeAt` far in the future) isolates the blocking schedule-loop cost from the immediate-fire flood.

## Candidate mitigations (for discussion)

- Move `UpdateTenantSchedules` re-scheduling off the host-blocking activation path (background it; allow readiness to proceed), or batch/throttle it.
- Throttle/jitter catch-up of past-due timers (bounded concurrency) instead of firing all at once at boot.
- Page the trigger/bookmark queries rather than loading everything into memory.

Related to #7732 (shares the underlying orphan-accumulation churn; both can contribute to the same slow-startup/crash-loop symptom).

["bug", "performance"]

Start by reading ActivateTenants.cs, UpdateTenantSchedules.cs, DefaultBookmarkScheduler.cs, and ScheduledSpecificInstantTask.cs, then run the suggested EFCore reproduction. Compare empty, future-dated, and past-due bookmark databases to separate scheduling-loop cost from the immediate-fire flood. Done should be a measured confirmation of the startup impact and an agreed mitigation scope.

Written by the indexing model from the issue text.