Statistics collection can wedge permanently: no timeout on layered query chains and a one-way in-flight flag
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### What happened?
Coordinator statistics collection has no timeout and a one-way in-flight flag, so a single lost `queryStatistics` reply can permanently wedge region advancement and permanently degrade stats — with no error ever surfaced.
Three compounding defects in `QueryWorkerStatisticsHandler.scala`:
1. **No timeout on layered query chains.** The handler traverses the sender's upstream cone layer by layer, and each layer awaits all replies before the next is emitted: `Future.collect(futures).flatMap(_ => processLayers(rest))` (:171) — no timeout, no fallback. `PortCompletedHandler` chains **all** port-completion bookkeeping (including region completion / scheduling the next region) on that future, so one dropped reply-expecting `queryStatistics` message wedges region advancement forever.
2. **`globalQueryStatsOngoing` is set optimistically and cleared only on success.** It is set to `true` at :101 when a full-graph query starts and cleared only inside the success `.map` (:183-185). If the chain never resolves, every later full-graph query short-circuits to the stale-cache branch (:86-91) for the life of the execution — statistics silently freeze.
3. **Dead workers keep getting queried.** `ExecutionUtils.aggregateStates` returns `COMPLETED` only when **all** worker states are COMPLETED or **all** are TERMINATED (:95-96); a mixed COMPLETED/TERMINATED set falls through to `UNKNOWN` (:98-107), so the operator is not skipped by the completed-operator check and its already-stopped workers are queried again — feeding defect 1, since those replies may never come. Worker states are additionally set to TERMINATED one at a time from Pekko dispatcher threads during teardown (`RegionExecutionManager.scala:214-219`), concurrent with these coordinator-thread reads.
```
lost queryStatistics reply
└─> layer's Future.collect never resolves (defect 1)
├─> portCompleted continuation never runs -> region advancement wedged
└─> globalQueryStatsOngoing stuck true (defect 2)
└─> all later stats polls serve stale cache
```
Suggested direction: a `.within(...)` per layer with a logged fallback (treat missing metrics as stale rather than blocking), clear `globalQueryStatsOngoing` in a `transform`/`ensure` rather than `map`, and make the mixed COMPLETED/TERMINATED aggregate count as completed for the skip check.
### How to reproduce?
Any path that loses a single worker→coordinator `queryStatistics` reply (e.g. a worker stopped between EndWorker acknowledgment and a late-layer stats query during teardown — reachable, see the analysis in #6916). The execution then hangs with no ERROR log, and stats stop updating.
### Version/Branch
main (observed at 429be110a7; discovered during the investigation for #6916).
Contributor guide
Research direction
Start with QueryWorkerStatisticsHandler.scala around lines 86-101 and 171-185, then inspect ExecutionUtils.aggregateStates and RegionExecutionManager.scala lines 214-219. Trace the layered query and port-completion paths, including teardown races. Done means lost replies no longer wedge advancement, the in-flight flag is cleared after failure, and mixed completed/terminated workers are skipped appropriately, with fallback logging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100