Region-termination retry budget is bypassed by non-delivery, wedging the workflow silently
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### What happened?
The 150-attempt region-termination retry budget only bounds **failures** — it is bypassed entirely by **non-delivery**, so a single failed `gracefulStop` pass can wedge the whole workflow silently, with the give-up ERROR never firing. This is exactly the silent-hang class that the bounded retry (#5737) was meant to eliminate.
Sequence:
```
attempt K: terminateWorkers
├─ endWorker OK for all workers
├─ for each worker: removeActorRef (:198), removeControlChannel x2 (:201-202) <- state already torn down
└─ gracefulStop fails for ONE worker (e.g. a slow Python PVM > 5 s) <- attempt K fails
attempt K+1: re-sends endWorker to ALL workers (RegionExecutionManager.scala:177-184)
├─ removed worker, ref gone -> message parks FOREVER in messageStash (PekkoActorRefMappingService.scala:69-73)
└─ ref still resolvable -> coordinator seq was reset to 0 by removeControlChannel
(NetworkOutputGateway.scala:97-101) while the worker's channel
`current` is still > 0 -> classified duplicate and silently
dropped (AmberFIFOChannel.scala:60-61; debug-only log at :46-49)
either way: Future.collect(endWorkerRequests) (:186-187) never resolves
-> rescue never runs -> retry budget never consumed -> give-up ERROR (:237) never fires
-> terminationFutureRef (CAS-set once, :157-171) wedges advanceRegionExecutions
(WorkflowExecutionManager.scala:108-113) -> no later region ever starts
```
Aggravating details:
- There is **no timeout anywhere** on the path: not on the `endWorker` collect, not on AsyncRPCClient promises (a repo grep for `.within(|raiseWithin|TimeoutException` in amber main sources hits only unrelated files).
- `actorRefService.getActorRef(workerId)` at `RegionExecutionManager.scala:196` is a bare map apply (`PekkoActorRefMappingService.scala:50-52`) and throws `NoSuchElementException` for a worker whose ref was removed by a previous attempt.
- The 30 s resend timer cannot rescue the seq-reset case: it re-sends the same (now duplicate) sequence number forever. Its suppression guard is separately broken (filed separately).
A per-attempt `.within(...)` timeout on the `endWorker`/`gracefulStop` collects (using the existing `killRetryTimer`) would make the budget honest; the channel/ref teardown at :198-202 probably needs to move after the *whole region's* stop pass succeeds, not run per-attempt.
### How to reproduce?
Make one worker's `gracefulStop` exceed the 5 s bound during region termination (e.g. a Python worker whose PVM is slow to exit) on a multi-worker region. Attempt 2's `endWorker`s either stash forever or are dropped as duplicates; the workflow hangs with no ERROR log.
### Version/Branch
main (observed at 429be110a7; discovered during the investigation for #6916).
Contributor guide
Assessment
This issue has not been assessed yet.