apache / apache/texera

LoopIntegrationSpec workflows intermittently hang on the macOS CI runner until the harness deadline

Open
#7,056 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### What happened?

On the macOS `amber-integration` job, a `LoopIntegrationSpec` workflow intermittently stops making progress mid-run with no error logged, burns the harness completion deadline, and is re-run by `withRetry` (failing the suite when both attempts hang). Observed in 3 of 4 attempts across two runs on 2026-07-29; never observed on ubuntu.

Log signature (job log, WARN backstop level): steady Python-worker spawns every ~2s, then total silence for ~the full deadline, then the retry's spawns resume. In the [failed run](https://github.com/apache/texera/actions/runs/30479437487/job/90673007976) an `EndHandler` `worker still has unprocessed messages` race (recovered ~23s before the silence) appears nearby, but the same race also self-heals in passing runs and locally, so it is not confirmed as the cause. `RegionExecutionManager` region-termination retries never log a second attempt during the silence, pointing at an await with no timeout between iterations (e.g. a termination or scheduling future that never resolves).

Expected: a healthy loop workflow completes in well under a minute; a broken one fails with a diagnosable error instead of silently idling.

### How to reproduce?

Re-run `build / amber-integration (macos-latest, 17)` on any PR until it exceeds ~13 min; the suite hangs most runs (e.g. runs 30479437487 and 30478651425 attempt 2 on 2026-07-29). Does not reproduce locally on an M-series Mac (JDK 17, venv with amber requirements, regenerated proto bindings): 4/4 pass in ~1.5 min with zero hangs, so the window is likely specific to the 3-core CI runner's timing. #7053 trims the suite and deadline so each hang costs ~2 min instead of ~4.

### Version/Branch

1.3.0-incubating-SNAPSHOT (main)

### Commit Hash (Optional)

_No response_

### What browsers are you seeing the problem on?

_No response_

### Relevant log output

```shell
18:45:19 WARN [WF6-LimitOpDesc-...-main-0] Received EndHandler before all messages are processed. Unprocessed: ControlInvocation(queryStatistics, ...)
18:45:19 ERROR [COORDINATOR] AsyncRPCClient - received error from Worker:WF6-LimitOpDesc-...
18:45:19 WARN [COORDINATOR] RegionExecutionManager - Failed to terminate region 1 on attempt 1 of 150. Retrying in 200 ms.
18:45:42
```

Contributor guide

Open the contributing guide

Research direction

Start by running the amber-integration job on macos-latest and reading LoopIntegrationSpec with its withRetry deadline behavior. Trace RegionExecutionManager termination retries and the EndHandler unprocessed-message warning around the reported log sequence, then identify the await that can remain unresolved. Done means the workflow no longer silently idles and reports a diagnosable failure when progress stops.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, macos, python, scala
Domain
ci-cd, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.