jenkinsci / jenkinsci/google-compute-engine-plugin

Launch timeout is applied 1000x longer than configured (seconds passed as milliseconds, multiplied again)

Open
#560 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
58
Forks
97
PR merge metrics
No merged PRs in 30d

Description

### Jenkins and plugins versions report

- Jenkins: 2.520+ (observed on current LTS line)
- google-compute-engine: 4.815.v9a_de97053e3a_ (latest; bug present at least since the `launchTimeout` wiring was introduced)

### What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Linux (k8s). Agents: GCE VMs, Debian 13.

### Reproduction steps

1. Configure a GCE cloud with an instance configuration, launch timeout = 300 seconds.
2. Provision an agent whose GCE insert operation ends in an error the plugin cannot observe quickly (e.g. the operation expires and `getZoneOperation` starts returning 404, or SSH never comes up).
3. Watch the launch thread: it does not give up after 300 seconds.

### Expected Results

Launch aborts after the configured 300 seconds (`launchTimeoutSeconds`), the node is terminated, and the thread exits.

### Actual Results

The launch thread waits **1000× the configured timeout** — a 300-second timeout becomes 300,000 seconds (~3.5 days). In our production instance we found 16 `Computer.threadPoolForRemoting` threads wedged for a week polling GCE operations from days earlier (each poll returning `404 Not Found` because the zone operation had long expired), with the corresponding nodes stuck offline in "connecting" state and their VMs kept alive indefinitely by `CleanLostNodesWork` label refreshes:

```
THREAD: Computer.threadPoolForRemoting [#64] state=TIMED_WAITING
...
com.google.cloud.graphite.platforms.plugin.client.ComputeClient.waitForOperationCompletion(ComputeClient.java:669)
com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:195)
hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
```

```
com.google.cloud.graphite.platforms.plugin.client.ComputeClient WARNING Error retrieving operation.
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
GET https://compute.googleapis.com/compute/v1/projects//zones/us-east1-b/operations/operation-1782936695633-...
```

The launcher's own log message shows the wrong value being used:

```
INFO: Launch will wait 300000 for operation operation-... to complete...
```

### Root cause

A units mismatch when the node is built:

- `InstanceConfiguration.provision()` passes `getLaunchTimeoutMillis()` (seconds × 1000) into the node builder: [`InstanceConfiguration.java` `.launchTimeout(getLaunchTimeoutMillis())`](https://github.com/jenkinsci/google-compute-engine-plugin/blob/9ade97053e3a535e1ddf810ec8d44a178118371f/src/main/java/com/google/jenkins/plugins/computeengine/InstanceConfiguration.java#L467)
- but `ComputeEngineInstance.launchTimeout` is documented and consumed as **seconds**: [`ComputeEngineInstance.java` `private Integer launchTimeout; // Seconds`](https://github.com/jenkinsci/google-compute-engine-plugin/blob/9ade97053e3a535e1ddf810ec8d44a178118371f/src/main/java/com/google/jenkins/plugins/computeengine/ComputeEngineInstance.java#L64) and `getLaunchTimeoutMillis()` multiplies by 1000 again.

So every wait that uses `node.getLaunchTimeoutMillis()` (operation wait, startup-script wait, SSH wait) runs with a timeout 1000× larger than configured.

### Anything else?

Knock-on effects we observed: `OnceRetentionStrategy.check` keeps calling `connect()` on the stuck node, wedging an additional thread per retention cycle; the stuck registered node keeps `CleanLostNodesWork` refreshing the VM's `jenkins_node_last_refresh` label, so the orphaned VM is never reaped and burns quota/cost until manual cleanup.

### Are you interested in contributing a fix?

Yes — PR incoming (one-line fix plus a regression test).

Contributor guide

Open the contributing guide

Research direction

Start with InstanceConfiguration.java at the launchTimeout wiring, then compare ComputeEngineInstance.java with the timeout use in ComputeEngineComputerLauncher.java and ComputeClient.java. Review the regression-test coverage mentioned in the issue. Done means a configured 300-second timeout is treated as 300 seconds across operation, startup-script, and SSH waits, with the regression test passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.