LoopIntegrationSpec workflows intermittently hang on the macOS CI runner until the harness deadline
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### What happened?
On the macOS `amber-integration` job, a `LoopIntegrationSpec` workflow intermittently stops making progress mid-run with no error logged, burns the harness completion deadline, and is re-run by `withRetry` (failing the suite when both attempts hang). Observed in 3 of 4 attempts across two runs on 2026-07-29; never observed on ubuntu.
Log signature (job log, WARN backstop level): steady Python-worker spawns every ~2s, then total silence for ~the full deadline, then the retry's spawns resume. In the [failed run](https://github.com/apache/texera/actions/runs/30479437487/job/90673007976) an `EndHandler` `worker still has unprocessed messages` race (recovered ~23s before the silence) appears nearby, but the same race also self-heals in passing runs and locally, so it is not confirmed as the cause. `RegionExecutionManager` region-termination retries never log a second attempt during the silence, pointing at an await with no timeout between iterations (e.g. a termination or scheduling future that never resolves).
Expected: a healthy loop workflow completes in well under a minute; a broken one fails with a diagnosable error instead of silently idling.
### How to reproduce?
Re-run `build / amber-integration (macos-latest, 17)` on any PR until it exceeds ~13 min; the suite hangs most runs (e.g. runs 30479437487 and 30478651425 attempt 2 on 2026-07-29). Does not reproduce locally on an M-series Mac (JDK 17, venv with amber requirements, regenerated proto bindings): 4/4 pass in ~1.5 min with zero hangs, so the window is likely specific to the 3-core CI runner's timing. #7053 trims the suite and deadline so each hang costs ~2 min instead of ~4.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
18:45:19 WARN [WF6-LimitOpDesc-...-main-0] Received EndHandler before all messages are processed. Unprocessed: ControlInvocation(queryStatistics, ...)
18:45:19 ERROR [COORDINATOR] AsyncRPCClient - received error from Worker:WF6-LimitOpDesc-...
18:45:19 WARN [COORDINATOR] RegionExecutionManager - Failed to terminate region 1 on attempt 1 of 150. Retrying in 200 ms.
18:45:42
```
Contributor guide
Research direction
Start by running the amber-integration job on macos-latest and reading LoopIntegrationSpec with its withRetry deadline behavior. Trace RegionExecutionManager termination retries and the EndHandler unprocessed-message warning around the reported log sequence, then identify the await that can remain unresolved. Done means the workflow no longer silently idles and reports a diagnosable failure when progress stops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, macos, python, scala
- Domain
- ci-cd, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100