jenkinsci / jenkinsci/openstack-cloud-plugin

JCloudsCleanupThread stalls with HTTP/2 REFUSED_STREAM (OVH Public Cloud); failed termination resets the idle clock, making agents immortal

Open
#475 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
49
Forks
97
Avg merge
13h 23m
Merged PRs (30d)
6

Description

## Environment

- Jenkins core 2.576 (official container image, K3D/Kubernetes)
- openstack-cloud 1465.vd3b_a_103e31fd (latest), okhttp-api 5.3.2-200.vedb_720a_cf1f8 (latest)
- Provider: OVH Public Cloud (Keystone v3 at https://auth.cloud.ovh.net/v3/, region GRA11, Nova at compute.gra11.cloud.ovh.net)
- One cloud, one template, JNLP launcher, `retentionTime: 30`, `instanceCap: 3`

## Symptoms

1. The controller log fills with `OpenStack slave cleanup thread is still running. Execution aborted.` — bursts of 13-19 entries over ~30 min windows, so each cleanup pass runs long enough to collide with the next tick.
2. Passes regularly die with:

```
SEVERE j.p.o.c.JCloudsCleanupThread#execute: Unable to perform the cleanup
org.openstack4j.api.exceptions.ConnectionException: stream was reset: REFUSED_STREAM
caused by okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:166)
at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:105)
...
```

3. Net effect: **ephemeral agents are never reaped**. We had idle agents alive (and billing) for 48h+ with `retentionTime: 30`.

## The part that makes it self-sustaining

While watching an idle agent that should have been reaped, its `Computer.getIdleStartMilliseconds()` jumped forward every ~30-36 minutes with **zero builds on the node** (checked per-executor: both executors' `idleStart` reset, at slightly different times). That matches the retention period: every time the retention check decides to terminate and the termination path fails on the OpenStack API, the node's executors appear to be recycled, which resets the idle clock — so the next attempt is again 30 minutes away, forever. The agent is effectively immortal.

## Ruled out

- The OVH API itself is healthy: from the controller host, Keystone answers 200 in ~55 ms consistently; Nova responds normally. Provisioning (few API calls) works fine throughout — only the chatty cleanup path degrades, which is what points at HTTP/2 stream exhaustion on a long-lived connection (OVH's API gateway refusing new streams, `REFUSED_STREAM`).
- A full controller restart does not help: the very first cleanup pass on the fresh JVM already collides with the next tick, and the SEVERE reappears within the hour.
- Plugin and okhttp-api are the latest released versions.

## Workaround

`Jenkins.get().getComputer(name).deleteSlave()` from the script console works instantly and reliably (node + Nova server cleanly destroyed), which confirms single destroy calls are fine — it is the bulk cleanup/listing burst that chokes.

## Suggestions

- Retrying `REFUSED_STREAM` on a fresh connection (or forcing HTTP/1.1 for the openstack4j connector, or capping concurrent streams) would likely fix the SEVERE.
- Independently, a failed termination should not reset the node's idle accounting — that turns a transient API failure into permanently leaked (billed) instances.

Happy to provide more logs or test a patched build against OVH.

Contributor guide

Open the contributing guide

Research direction

Start at JCloudsCleanupThread#execute and trace the retention termination path, using the reported REFUSED_STREAM stack trace and idleStart resets as the reproduction signals. Compare the bulk cleanup/listing flow with the reliable deleteSlave() script-console path. Done means cleanup no longer stalls on refused HTTP/2 streams and a failed termination does not make the agent's idle countdown restart indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.