jenkinsci / jenkinsci/google-compute-engine-plugin
Large number of offline build executors when there is no capacity
- Dominant language
- Java
- Stars
- 58
- Forks
- 97
- PR merge metrics
- No merged PRs in 30d
Description
### Jenkins and plugins versions report
Environment
```text
google-compute-engine:4.3.14
google-oauth-plugin:1.0.8
```
### What Operating System are you using (both controller, and any agents involved in the problem)?
Linux AMD64 - Ubuntu
### Reproduction steps
1. Create a template for e.g. a GPU node, which are in short supply
2. Select a region with no capacity left
3. Try to run a job
4. See up to 20 build executors created, all in the "offline" state
```
Started provisioning jenkins-worker-4-a100-gpu-b-q4vmv1 from gce-GCE with 1 executors. Remaining excess workload: 0
Sep 26, 2023 4:36:02 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud lambda$getPlannedNodeFuture$0
Waiting 300000ms for node jenkins-worker-4-a100-gpu-b-q4vmv1 to connect
Sep 26, 2023 4:36:12 PM INFO com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher launch
Launch failed while waiting for operation operation-1695738961191-60643fe69e3e8-bcfd7998-b4900d7d to complete. Operation error was The zone 'projects/data-fabric-272407/zones/europe-west4-b' does not have enough resources available to fulfill the request. '(resource type:compute)'.
Sep 26, 2023 4:36:19 PM WARNING hudson.slaves.NodeProvisioner update
Unexpected exception encountered while provisioning agent jenkins-worker-4-a100-gpu-b-q4vmv1
java.io.IOException: Agent failed to connect, even though the launcher didn't report it. See the log output for details.
at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:326)
Caused: java.util.concurrent.ExecutionException
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
at com.google.jenkins.plugins.computeengine.ComputeEngineCloud.lambda$getPlannedNodeFuture$0(ComputeEngineCloud.java:315)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
### Expected Results
The plugin should recognize that the launch of the node failed (no capacity), and not show it in the list of build executors at all.
### Actual Results
A list of 20 build executors, all offline.
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.