jenkinsci / jenkinsci/google-compute-engine-plugin

Compute engine nodes stuck in endless "Instance <instance-name> is being shut down..." loop

Open
#567 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
58
Forks
97
PR merge metrics
No merged PRs in 30d

Description

### Jenkins and plugins versions report

Google Compute Engine nodes (lauched with this plugin) will sometimes get stuck in an endless loop of following in the node log:
```
FINEST: Instance is being shut down...
```

I have a node that's currently impacted, and it's state shows as `TERMINATED`:
```
% gcloud compute instances describe jenkins-ubuntu-zjxc5k --format="value(status)"
TERMINATED
```

Environment


```text
Jenkins: 2.541.3
OS: Linux - 5.14.0-503.40.1.el9_5.cloud.1.0.x86_64
Java: 21.0.11 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
ansicolor:536.v13fa_b_860c267
ant:520.vd082ecfb_16a_9
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
asm-api:9.10.1-216.va_9256d3b_844b_
atlassian-jira-software-cloud:2.0.15
authentication-tokens:1.144.v5ff4a_5ec5c33
basic-branch-build-strategies:317.v85b_331d6cc42
bootstrap5-api:5.3.8-895.v4d0d8e47fea_d
bouncycastle-api:2.30.1.84-291.v9f17b_21896e2
branch-api:2.1280.v0d4e5b_b_460ef
build-timeout:1.40
caffeine-api:3.2.4-208.v7e2da_a_7db_82b_
checks-api:415.vf022234a_931d
cloudbees-folder:6.1100.ve9eed61d16c4
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-218.va_61573470393
credentials:1502.v5c95e620ddfe
credentials-binding:725.ve52b_2328a_fde
dark-theme:652.vea_da_dfea_e769
data-tables-api:2.3.7-1534.v539d4edf109d
declarative-pipeline-migration-assistant:1.6.6
declarative-pipeline-migration-assistant-api:1.6.6
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:472.vee120e23d3a_c
docker-workflow:634.vedc7242b_eda_7
durable-task:671.v340ff7959010
echarts-api:6.0.0-1247.vf3e35a_c1813f
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
email-ext:2038.v7b_8817a_499d9
font-awesome-api:7.2.0-965.ve3840b_696418
git:5.10.1
git-client:6.6.1
github:1.47.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1967.1970.vd86979736546
github-checks:679.v74133da_b_435a_
github-scm-trait-notification-context:45.v8ef831829589
google-compute-engine:4.815.v9a_de97053e3a_
google-oauth-plugin:1.346.v712b_a_e0d1e60
gradle:2.19.1244.v1f9866817fec
gson-api:2.14.0-201.v8eefe5515533
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson-annotations2-api:2.22-19.v10a_a_582ea_26e
jackson2-api:2.22.0-439.vb_b_43658d6176
jackson3-api:3.2.0-83.vfb_04cee1ea_0f
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.9-19.v2b_a_5b_44d9a_1c
javax-activation-api:1.2.0-8
jaxb:2.3.9-143.v5979df3304e6
jjwt-api:0.13.0-141.vd58b_a_9592b_6c
job-restrictions:242.v6edda_c9e4ca_f
joda-time-api:2.14.2-193.v422b_efce56e0
jquery3-api:3.7.1-619.vdb_10e002501a_
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20260522-217.v0b_18b_8cd4672
json-path-api:3.0.0-218.vcd4dd1355de2
jsoup:1.22.2-95.vc5d00f1eb_42d
junit:1413.v736fa_5b_61d80
ldap:807.809.vd3a_4e5e4ec98
lockable-resources:1537.v68d014129e68
mailer:534.v1b_36f5864073
matrix-auth:3.2.10
matrix-project:870.v9db_fcfc2f45b_
metrics:4.2.37-494.v06f9a_939d33a_
mina-sshd-api-common:2.17.1-187.v0341274c2905
mina-sshd-api-core:2.17.1-187.v0341274c2905
oauth-credentials:0.660.vce7c108f1f3a_
okhttp-api:5.3.2-200.vedb_720a_cf1f8
pam-auth:1.12
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:254.v0f63a_a_447dca_
pipeline-graph-view:953.v23894ccf0079
pipeline-groovy-lib:798.v5cc688825312
pipeline-input-step:551.vdff487c5998c
pipeline-milestone-step:152.v6e22b_8cfc66c
pipeline-model-api:2.2291.v2934911987b_6
pipeline-model-definition:2.2291.v2934911987b_6
pipeline-model-extensions:2.2291.v2934911987b_6
pipeline-rest-api:2.41
pipeline-stage-step:345.va_96187909426
pipeline-stage-tags-metadata:2.2291.v2934911987b_6
pipeline-stage-view:2.41
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1192.v30fe6e2837ff
prism-api:1.30.0-717.vb_f8360844b_53
resource-disposer:0.25
scm-api:728.vc30dcf7a_0df5
script-security:1402.1405.vc96e74964250
snakeyaml-api:2.5-149.v72471e9c6371
snakeyaml-engine-api:3.0.1-5.vd98ea_ff3b_92e
ssh-agent:396.vcc7d84e622ec
ssh-credentials:372.va_250881b_08cd
ssh-slaves:3.1097.v868116049892
structs:362.va_b_695ef4fdf9
theme-manager:346.v06cca_64c6a_37
throttle-concurrents:625.vc8b_e469e9a_b_c
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
variant:70.va_d9f17f859e0
woodstox-core-api:7.2.1-6.v3718a_a_11f5c4
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1413.v2ff1a_5e720fa_
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4331.4333.v50a_b_076c5199
workflow-durable-task-step:1479.v56e587f413a_7
workflow-job:1571.1580.v18e46842c125
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:724.v538c2362b_dfb_
workflow-support:1015.v785e5a_b_b_8b_22
ws-cleanup:0.49
```

### What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Rocky Linux 9.7 (Blue Onyx)
Agent: Ubuntu 24.04.4 LTS (Noble Numbat)

### Reproduction steps

I cannot reliably reproduce this issue, but it occurs frequently in my environment. (At least once per week.)

### Expected Results

Node is deleted after it shuts down.

### Actual Results

Node is stuck in the state described where the Jenkins log for the node keeps repeating:
```
FINEST: Instance is being shut down...
```

### Anything else?

_No response_

### Are you interested in contributing a fix?

I have a potential fix. Would like the maintainers to review.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Google Compute Engine plugin's node shutdown handling around the repeated `Instance is being shut down...` log, and use `gcloud compute instances describe ... --format="value(status)"` to confirm the instance is TERMINATED. Done means a shut-down node is deleted instead of remaining in the loop; the issue notes that reproduction is intermittent.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.