apache / apache/grails-core

CI: the whole-repository build hangs after its last task, holding a runner until the 6 hour kill

Open
#16,218 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Groovy
Stars
2.9k
Forks
975
Avg merge
1d 22h
Merged PRs (30d)
92

Description

The job named `Neo4j Functional Tests (Java 25, indy=false)` stops producing output after its last task and never finishes, holding a runner until GitHub kills it at the six hour ceiling. Seen twice. All tests in the job have passed by the time it stops.

Despite the name, nothing about this is Neo4j. That job runs

```
./gradlew bootJar check --continue --rerun-tasks -PgrailsIndy=... -PonlyNeo4jTests -PskipCodeStyle
```

which is a whole-repository build; `-PonlyNeo4jTests` gates only the functional example applications, so the module tests of the whole repository run in it as well.

## Occurrences

| Date (UTC) | Branch | Job | Outcome |
|---|---|---|---|
| 2026-08-22 18:34 | `fix/embedded-mongo-replica-set` | [attempt 1](https://github.com/apache/grails-core/actions/runs/32591084544/job/97074994493) | ran 18:34:00 → 00:34:16, **exactly 6h00m**, cancelled by the platform |
| 2026-08-22 (retry) | same | attempt 2 | hung again, cancelled manually after 2h18m |
| 2026-08-24 22:49 | `ci/release-drafter-pr-concurrency` (#16213) | [job](https://github.com/apache/grails-core/actions/runs/32786454527/job/97619273551) | hung 4h44m, cancelled manually |

## Where it stops

Identical in both, to the line. The last thing in the log is the AOT example application starting and shutting down, then its `check` task, and then nothing:

```
INFO 50196 --- [main] o.s.boot.tomcat.TomcatWebServer : Tomcat started on port 46097 (http)
INFO 50196 --- [main] aot.Application : Started Application in 7.911 seconds
Grails application running at http://localhost:46097 in environment: production
INFO 50196 --- [main] o.s.boot.tomcat.GracefulShutdown: Commencing graceful shutdown
INFO 50196 --- [tomcat-shutdown] o.s.boot.tomcat.GracefulShutdown: Graceful shutdown complete

> Task :grails-test-examples-aot:check
← 219 minutes of silence →
##[error]The operation was canceled.
```

In a run that succeeds, `:grails-test-examples-aot:check` is also the last task, and the next lines follow immediately:

```
> Task :grails-test-examples-aot:check
gradle/actions: Writing build results to /home/runner/work/_temp/.gradle-actions/build-results/...json
[Incubating] Problems report is available at: ...
BUILD SUCCESSFUL in 1h 1m 30s
```

So every task has completed and the build stops before Gradle reports its result. No test is running at that moment; the AOT application, which logs its own pid, has already exited.

## Processes alive when the runner cleaned up

| Run | orphan processes | of which `mongod` |
|---|---|---|
| 2026-08-22 | 22 | 3 |
| 2026-08-24 | 18 | 0 |

The three mongods in the first were embedded MongoDB servers left running by specifications that relied on the JVM shutdown hook to reap them. That was a genuine leak and it has since been fixed (`8dca5f2f`, in #16192) - the second run shows none. **It did not fix this hang**, which is the reason for this issue: the leak and the hang are separate problems, and only the leak is addressed.

None of the java pids in either orphan list appears anywhere else in the log, so they are the Gradle daemon and its test workers rather than application JVMs. Gradle keeps idle workers for reuse during a build and stops them when the build ends, so their presence at the kill is consistent with a build that never ended and does not by itself say they caused it.

## What is not known

What the build is doing during the silence. Two data points, no thread dump, and no reproduction: the same job passes on the same branches at other times, taking 40-65 minutes, which is its normal range on `8.0.x`.

One thing worth noting rather than concluding from: `develocity.apache.org` returns `403 Forbidden` on cache stores throughout both hung runs

```
java.lang.RuntimeException: Storing entry at 'https://develocity.apache.org/cache/...' response status 403: Forbidden
at com.gradle.develocity.agent.gradle.internal.a.c.store(SourceFile:69)
```

and on 2026-08-24 a different job failed outright because the Develocity plugin could not be resolved at all. The build's post-task phase talks to that endpoint. That makes it a candidate, not an explanation.

## Suggested next step

A thread dump would end the guessing rather than add to it. A step on this job that waits N minutes and then runs `jstack` against the Gradle daemon and its workers, printing the result, would name the blocked thread on the next occurrence. Something along the lines of:

```yaml
- name: "Dump threads if the build is still running"
if: always()
run: |
for pid in $(jps -q); do
echo "=== $pid"
jstack "$pid" || true
done
```

run from a background watchdog started before the build, firing once the build passes its normal ceiling.

## Other CI flakes tracked separately

- #16219 - `AsyncPromiseSpec` times out waiting for a response. It surfaces in the same kind of whole-repository build as this hang, and the runs of both carry the same `develocity.apache.org … 403: Forbidden` noise. Whether that is a common cause or a common environment is not established.
- #16217 - the scaffolding application's login form is submitted without a session cookie. No known relationship to this one.

Contributor guide

Open the contributing guide

Research direction

Start with the GitHub Actions definition for `Neo4j Functional Tests (Java 25, indy=false)` and run its whole-repository Gradle command to understand the normal completion path. Add the suggested watchdog around the build, then inspect the Gradle daemon and worker thread dumps on a recurrence; done means the blocked thread or post-task operation is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, groovy
Domain
build-system, ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.