[Bug]: Entity is never redispatched after a worker disconnect; the critical-section lock it holds leaks permanently (Dedicated SKU, .NET isolated)

> **Note on where this is filed.** This is a **service/backend** issue, not a sample issue. The
> `Azure/Durable-Task-Scheduler` repo — which this repo's issue-template `config.yml` still points to
> for service issues — now displays a banner stating it is no longer monitored and redirects here.
> Filing here on that basis, and because #364 and #331 were service issues handled here. Happy to
> move it if that is wrong.

### Which sample is affected?

Durable Functions - .NET

### Describe the bug

When a worker disconnects while an **entity** work item is in flight, that entity is never dispatched
again. Messages sent to it afterwards — including from a fresh, unrelated client — accumulate in its
`BacklogQueueSize` and are never processed, indefinitely (observed 4+ hours, and 24 h in one run).

Any orchestration holding a critical-section lock over that entity is then parked forever, and its
locks are never released, because `await using (await LockEntitiesAsync(...))` only disposes when the
critical-section body completes — and the body can never complete while one entity in the lock set is
not answering. **The leaked lock is the symptom; the entity leaving the dispatch loop is the cause.**
Terminating the lock holder does not release the lock, because the orchestration is not what is
wedged.

**Orchestration and activity work items abandoned in the same bursts ARE redelivered. Entity work
items are not.** The host's own drain message states the expectation:

```
Worker drain timed out after 59788.0211 ms. 1 work item(s) were still in-flight and will be
abandoned by the backend; they will be redelivered to other workers.
```

There is no error surfaced anywhere: no worker exception, no failed orchestration, no platform health
signal. The orchestration simply stays `Running` forever. **This failure is silent by construction.**

#### Scaling the scheduler does not fix it

Same workload, same scheduler resource, at two capacities:

| run | scheduler | duration | stuck orchestrations |
|---|---|---|---|
| A | Dedicated **1 CU** | ~6 h | 243 |
| B | Dedicated **1 CU** | ~3.5 h | 434 (a floor) |
| C | Dedicated **1 CU**, two apps / two hubs | ~22.75 h | 461 |
| **D** | **Dedicated 2 CU**, two apps / two hubs | **~14.5 h** | **312** |

Run D existed specifically to rule out scheduler saturation, since run C reached its action ceiling.
It did not help.

#### The failure has a hard timing signature

All 312 stuck orchestrations from run D, by how long each survived between creation and its last
recorded activity:

| survived before freezing | count |
|---|---|
| < 1 minute | **237** |
| 1 – 5 minutes | 75 |
| 5 – 30 minutes | **0** |
| > 30 minutes | **0** |

Not one froze after the 5-minute mark, and not one has moved since — the shortest idle time across all
312 is over 30 minutes, the longest over 14 hours. This is not slow work hitting a timeout.

The two apps also froze in **disjoint windows** (one 06:00–09:00Z, the other 10:00–13:00Z) rather than
simultaneously, which points at per-app worker lifecycle rather than a shared backend load event.

#### Critical-section anatomy, from the orchestrations' own histories

Twelve stuck orchestrations sampled across run D, both hubs, spread across the whole run. Inside a
critical section each entity operation is an `EventSent` carrying an op id and each answer is an
`EventRaised` named with that id, so the anatomy is visible without touching the entity:

| | result |
|---|---|
| lock granted, then frozen mid-section | **9 of 12** |
| of those, with **exactly one** unanswered operation | **9 of 9** |
| of those, that ever sent a `release` message | **0 of 9** |
| no lock ever granted (a different shape, not analysed) | 3 of 12 |

Every frozen one is identical: lock granted, several operations sent and answered, then **one
operation sent and never answered**, the section never completes, `release` is never reached, the lock
is held forever.

**The drop position is random, which rules out a cold or uninitialized entity.** Operations answered
before the entity went silent, across the nine:

```
0 answered first : 2 4 answered first : 2
1 answered first : 1 5 answered first : 2
3 answered first : 1 6 answered first : 1
```

In seven of the nine, the entity had **already answered at least one operation from this same holder**
seconds earlier and then stopped. Four different operation names are represented, so it is not one
bad operation.

#### A matched pair, same minute, same hub

| | instance `06da2091…c565` | instance `69d2d6c0…ec3f` |
|---|---|---|
| lock acquired | 18:28:10Z, 4 entities | 18:29:18Z, 4 entities |
| ops in critical section | all answered | 5 answered, then **one at 18:32:23Z never answered** |
| `release` messages | **4 sent, 4 answered** | **never sent — never reached** |
| outcome | `Completed` 18:28:22Z (39 s end to end) | **`Running` 4 h later** |
| its 4 entities afterwards | `NOTFOUND` (released, state cleared) | **all 4 still locked by it**, one with `BacklogQueueSize = 6` |

The stuck one is not a cold or missing entity: the same entity answered two earlier operations from
the same holder seconds before, and its `lastModified` is frozen at that last answered operation. The
application also *won* its race — the awaited external event arrived inside its timeout — so the
application behaved correctly and the very next message was dropped. Entity state at freeze: **905
bytes** (limit 1 MB).

#### The scheduler's own metrics contradict its state

While >200 entities on the hub had `BacklogQueueSize > 0`, the scheduler reported:

- `EntityPendingItems` = **0**
- `EntityActiveItems` = **0**

The messages are persisted in the entities' queues, but the scheduler holds no work item for them and
never creates one. **It is not a lease that expires and retries — it is an entity that has left the
dispatch loop entirely.**

#### Completion rejections at the moment of the freeze

```
Status(StatusCode="NotFound", Detail="Work item '::'
with completion count '0' not found")
TaskEntityDispatcher-...: Unhandled exception with work item '@@'
Abandoning work item for entity '@@' with completion token ::
```

In sub-second bursts across many instances — 7 workers within 254 ms, and 29 workers producing 81
rejections within 12 s. **In those same bursts, orchestration work items were redelivered
successfully** (one rejected at 14:45:31 completed at 14:45:47); entity work items were not.

#### A client signal proves the entity is dead, not merely locked

A `SignalEntityAsync` from a **fresh external client a full day later**, to a frozen entity, was
counted in `BacklogQueueSize` (1 → 2) and never processed. A control entity signalled from the same
client in the same minute processed in **under 1 second**. The fleet and hub were healthy; only these
entities are dead.

#### What we ruled out on our side

| hypothesis | evidence against |
|---|---|
| We violate a critical-section rule | **0** `LockingRulesViolationException` in any run. Exactly one `LockEntitiesAsync` in the codebase (no nesting); the sub-orchestration call is before the lock, not inside; the only entity signalled inside the section is outside the lock set, which the docs permit. |
| Entity state exceeds 1 MB | 905 bytes on the frozen entity. |
| Instance id exceeds 100 chars / non-ASCII | All ids ≤ 100 and printable ASCII; we enforce both. |
| A worker-side entity failure we swallow | No exception logged for any frozen entity, by any worker, in any run. |
| Deadlock between holders | 0 of 243 waited on an entity held by another *running* holder that was itself waiting on them. Every chain terminates at a frozen entity. |
| Cancelled durable timers | 0 of 243 stuck orchestrations were awaiting a timer. |
| Scheduler under-provisioned | Reproduced at 2 CU after being found at 1 CU, with the identical signature. |
| Only one orchestration type affected | No — a second orchestration type taking a section over the same entities freezes identically (18 of 312 in run D). |

#### How this relates to #331 and #364

**This is not #331** (*Activities re-executed (~25%) on Elastic Premium scale-in: DrainMode drops the
DTS activity work-item lock*). That was fixed in
`Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureManaged` **1.9.0**; we run **1.10.0** and
already carry that fix.

The contrast with #331 is the clearest way to state this problem. There, the work item **was**
redelivered — too eagerly, causing duplicate execution. Here, entity work items are **never**
redelivered at all. In the very same churn bursts, on the same hubs, orchestration and activity work
items are redelivered and recover normally; only entities are lost. Whatever redelivery path #331
exercised for activities does not appear to exist for entities.

**This may share a trigger with #364** (*Dedicated-SKU scheduler suffers recurring backend cascades*),
where `ConnectedWorkers` collapses to 0 and workers see `NotFound` work items. We see that condition
too, and the team noted on 2026-07-22 that the causing bugs were identified with fixes in progress.

If those fixes land they may reduce how **often** workers are disconnected. They would not address
what this issue reports: that **when** a disconnect happens, an entity mid-work-item is orphaned
permanently, with no self-heal and no operator recovery path, while every other work-item type
recovers. That asymmetry is the defect, independent of how often the trigger occurs.

#### What we are asking

1. Is entity work-item redelivery after an abandoned/expired lease **implemented** the way it is for
orchestrations and activities? Our evidence says no.
2. Is there any way for a client or operator to **force redispatch** of an entity that has left the
dispatch loop? We have found none — signalling it does not, and terminating the lock holder does
not.
3. Why do `EntityPendingItems` / `EntityActiveItems` report **0** while entities hold non-empty
backlogs? If that is the same bug, the metric cannot be used to detect this.
4. Is there a supported way to detect this from outside? Today it is only visible by enumerating
entities looking for `BacklogQueueSize > 0` with a stale `LastModifiedTime`.
5. Separately: `EntityMetadata.LockedBy` returns only the first **50 characters** of a 64-character
holder instance id, which makes holders unresolvable without a prefix scan.

We can provide raw orchestration/entity ids, exact timestamps, and the scheduler resource id privately
via a support case — omitted here because this repo is public.

### Steps to reproduce

Deterministic under load on this configuration:

1. Durable Functions .NET isolated on Flex Consumption, DTS Dedicated SKU.
2. An orchestration that takes a critical section over ~4 entities and performs several sequential
entity calls inside it, holding the section across an `await` on an external event (seconds to
minutes).
3. Drive sustained load (~50k orchestrations/hour) so the fleet scales to its instance cap.
4. Force worker churn: `az functionapp restart`, a `config-zip` deploy, or simply let Flex recycle —
the host logs `Worker drain timed out … will be redelivered to other workers`.
5. Some entities that were mid-work-item at the disconnect never dispatch again. Their holders stay
`Running` forever and never release their locks.

Anything that briefly removes the scheduler's data plane reproduces it. In one run, **opening the DTS
dashboard against a hub under production load** caused a ~5-minute data-plane outage
(`ConnectedWorkers` 40 → 0) and produced 244 frozen entities by itself — that may warrant a separate
issue.

### Expected behavior

An entity work item abandoned by a disconnecting worker should be **redelivered to another worker**,
exactly as orchestration and activity work items are — as the host's own drain message states
("they will be redelivered to other workers").

Concretely, we would expect any of the following, none of which occur today:

- the entity resumes processing its backlog once a healthy worker is available;
- failing that, `EntityPendingItems` / `EntityActiveItems` reflect the non-empty backlog so the
condition is at least **detectable** and alertable;
- failing that, some supported operator action — purge, signal, or terminating the lock holder —
returns the entity to the dispatch loop.

Today the entity is permanently orphaned, every orchestration holding a lock on it is permanently
parked, and there is no signal and no recovery path short of moving to a new task hub.

### Environment

| | |
|---|---|
| Scheduler SKU | **Dedicated**, tested at **1 CU and 2 CU** (same resource, scaled), Central US |
| Host runtime | Azure Functions v4, `4.1053` |
| Plan | Flex Consumption, **512 MB**, max 40–50 instances, always-ready 1 |
| Worker | .NET 8 isolated |
| `Microsoft.Azure.Functions.Worker` | 2.52.0 |
| `Microsoft.Azure.Functions.Worker.Extensions.DurableTask` | 1.19.0 |
| `Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureManaged` | **1.10.0** (i.e. includes the #331 fix from 1.9.0) |
| `host.json` | `useGracefulShutdown: true`; `maxConcurrentOrchestratorFunctions` / `Activity` / `Entity` = 20; `maxEntityOperationBatchSize` = 100 |
| Load | 50k–60k orchestrations/hour, each taking a critical section over ~4 entities |
| Observed over | 4 runs, 5 task hubs, ~47 hours of production-volume traffic |

No source file or test is named. Start by reproducing the worker-churn scenario with Durable Functions .NET isolated, focusing on entity work-item dispatch after the worker drain message. Done means an abandoned entity is redelivered, its backlog is processed, and critical-section locks are released rather than remaining stuck.

Written by the indexing model from the issue text.