jenkinsci / jenkinsci/openstack-cloud-plugin
JCloudsCleanupThread stalls with HTTP/2 REFUSED_STREAM (OVH Public Cloud); failed termination resets the idle clock, making agents immortal
- Dominant language
- Java
- Stars
- 49
- Forks
- 97
- Avg merge
- 13h 23m
- Merged PRs (30d)
- 6
Description
## Environment
- Jenkins core 2.576 (official container image, K3D/Kubernetes)
- openstack-cloud 1465.vd3b_a_103e31fd (latest), okhttp-api 5.3.2-200.vedb_720a_cf1f8 (latest)
- Provider: OVH Public Cloud (Keystone v3 at https://auth.cloud.ovh.net/v3/, region GRA11, Nova at compute.gra11.cloud.ovh.net)
- One cloud, one template, JNLP launcher, `retentionTime: 30`, `instanceCap: 3`
## Symptoms
1. The controller log fills with `OpenStack slave cleanup thread is still running. Execution aborted.` — bursts of 13-19 entries over ~30 min windows, so each cleanup pass runs long enough to collide with the next tick.
2. Passes regularly die with:
```
SEVERE j.p.o.c.JCloudsCleanupThread#execute: Unable to perform the cleanup
org.openstack4j.api.exceptions.ConnectionException: stream was reset: REFUSED_STREAM
caused by okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:166)
at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:105)
...
```
3. Net effect: **ephemeral agents are never reaped**. We had idle agents alive (and billing) for 48h+ with `retentionTime: 30`.
## The part that makes it self-sustaining
While watching an idle agent that should have been reaped, its `Computer.getIdleStartMilliseconds()` jumped forward every ~30-36 minutes with **zero builds on the node** (checked per-executor: both executors' `idleStart` reset, at slightly different times). That matches the retention period: every time the retention check decides to terminate and the termination path fails on the OpenStack API, the node's executors appear to be recycled, which resets the idle clock — so the next attempt is again 30 minutes away, forever. The agent is effectively immortal.
## Ruled out
- The OVH API itself is healthy: from the controller host, Keystone answers 200 in ~55 ms consistently; Nova responds normally. Provisioning (few API calls) works fine throughout — only the chatty cleanup path degrades, which is what points at HTTP/2 stream exhaustion on a long-lived connection (OVH's API gateway refusing new streams, `REFUSED_STREAM`).
- A full controller restart does not help: the very first cleanup pass on the fresh JVM already collides with the next tick, and the SEVERE reappears within the hour.
- Plugin and okhttp-api are the latest released versions.
## Workaround
`Jenkins.get().getComputer(name).deleteSlave()` from the script console works instantly and reliably (node + Nova server cleanly destroyed), which confirms single destroy calls are fine — it is the bulk cleanup/listing burst that chokes.
## Suggestions
- Retrying `REFUSED_STREAM` on a fresh connection (or forcing HTTP/1.1 for the openstack4j connector, or capping concurrent streams) would likely fix the SEVERE.
- Independently, a failed termination should not reset the node's idle accounting — that turns a transient API failure into permanently leaked (billed) instances.
Happy to provide more logs or test a patched build against OVH.
Contributor guide
Research direction
Start at JCloudsCleanupThread#execute and trace the retention termination path, using the reported REFUSED_STREAM stack trace and idleStart resets as the reproduction signals. Compare the bulk cleanup/listing flow with the reliable deleteSlave() script-console path. Done means cleanup no longer stalls on refused HTTP/2 streams and a failed termination does not make the agent's idle countdown restart indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100