jenkinsci / jenkinsci/google-compute-engine-plugin

Jenkins slaves connection fails randomly

Open
#299 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
58
Forks
97
PR merge metrics
No merged PRs in 30d

Description

### Jenkins and plugins versions report

Environment


Jenkins: 2.327
OS: Linux - 5.4.0-1058-gcp
---
CustomHistory:1.6
ace-editor:1.1
ansicolor:1.0.1
ant:1.13
antisamy-markup-formatter:2.6
apache-httpcomponents-client-4-api:4.5.13-1.0
artifactory:3.14.2
authentication-tokens:1.4
authorize-project:1.4.0
aws-credentials:1.33
aws-java-sdk-ec2:1.12.131-302.vbef9650c6521
aws-java-sdk-minimal:1.12.131-302.vbef9650c6521
badge:1.9
bitbucket:214.v2fd4234d0554
bitbucket-push-and-pull-request:2.8.1
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.3-4
bouncycastle-api:2.25
branch-api:2.7.0
build-environment:1.7
build-monitor-plugin:1.13+build.202112271752
build-name-setter:2.2.0
build-pipeline-plugin:1.5.8
build-timeout:1.20
build-timestamp:1.0.3
build-user-vars-plugin:1.8
caffeine-api:2.9.2-29.v717aac953ff3
checks-api:1.7.2
chucknorris:1.4
cloudbees-folder:6.17
command-launcher:1.6
conditional-buildstep:1.4.1
config-file-provider:3.8.2
copyartifact:1.46.2
credentials:1055.v1346ba467ba1
credentials-binding:1.27
dashboard-view:2.18
display-url-api:2.3.5
docker-build-step:2.8
docker-commons:1.17
docker-java-api:3.1.5.2
docker-plugin:1.2.6
docker-workflow:1.26
durable-task:493.v195aefbb0ff2
echarts-api:5.2.2-2
email-ext:2.86
embeddable-build-status:2.0.3
extended-choice-parameter:0.82
extensible-choice-parameter:1.8.0
external-monitor-job:1.7
extra-columns:1.25
font-awesome-api:5.15.4-5
gcloud-sdk:0.0.3
generic-webhook-trigger:1.79
git:4.10.1
git-client:3.11.0
git-parameter:0.9.14
git-server:1.10
git-tag-message:1.7.1
github:1.34.1
github-api:1.301-378.v9807bd746da5
github-branch-source:2.11.4
google-compute-engine:4.3.8
google-container-registry-auth:0.3
google-kubernetes-engine:0.8.6
google-login:1.6
google-oauth-plugin:1.0.6
gradle:1.37.1
groovy-postbuild:2.5
handlebars:3.0.8
hidden-parameter:0.0.4
htmlpublisher:1.28
ivy:2.1
jackson2-api:2.13.1-244.v773c36c5b330
javadoc:1.6
jaxb:2.3.0
jdk-tool:1.5
jjwt-api:0.11.2-9.c8b45b8bb173
job-dsl:1.78.3
jobConfigHistory:2.31-rc1098.b666422863b2
jquery:1.12.4-1
jquery-detached:1.2.1
jquery-ui:1.0.2
jquery3-api:3.6.0-2
jsch:0.1.55.2
junit:1.53
kubernetes:1.31.1
kubernetes-client-api:5.10.1-171.vaa0774fb8c20
kubernetes-credentials:0.9.0
ldap:2.7
lockable-resources:2.13
mailer:1.34
matrix-auth:3.0
matrix-project:1.19
maven-plugin:3.16
mercurial:2.16
metrics:4.0.2.8
momentjs:1.1.1
nodejs:1.4.3
oauth-credentials:0.5
okhttp-api:4.9.3-105.vb96869f8ac3a
pam-auth:1.6.1
parameter-separator:1.3
parameterized-trigger:2.43
periodicbackup:1.7
pipeline-build-step:2.15
pipeline-github-lib:1.0
pipeline-graph-analysis:188.v3a01e7973f2c
pipeline-input-step:427.va6441fa17010
pipeline-milestone-step:1.3.2
pipeline-model-api:1.9.3
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:1.9.3
pipeline-model-extensions:1.9.3
pipeline-rest-api:2.20
pipeline-stage-step:291.vf0a8a7aeeb50
pipeline-stage-tags-metadata:1.9.3
pipeline-stage-view:2.20
pipeline-utility-steps:2.11.0
plain-credentials:1.7
plugin-util-api:2.10.0
popper-api:1.16.1-2
popper2-api:2.11.0-1
publish-over:0.22
publish-over-ssh:1.22
purge-job-history:1.6
pwauth:0.4
readonly-parameters:1.0.0
rebuild:1.32
resource-disposer:0.17
role-strategy:3.2.0
run-condition:1.5
saml:2.0.9
scm-api:2.6.5
script-security:1118.vba21ca2e3286
slack:2.49
snakeyaml-api:1.29.1
ssh:2.6.1
ssh-agent:1.23
ssh-credentials:1.19
ssh-slaves:1.33.0
sshd:3.1.0
stashNotifier:1.24
structs:308.v852b473a2b8c
summary_report:1.15
throttle-concurrents:2.6
timestamper:1.15
token-macro:267.vcdaea6462991
trilead-api:1.0.13
uno-choice:2.5.7
variant:1.4
view-job-filters:2.3
windows-slaves:1.8
workflow-aggregator:2.6
workflow-api:1108.v57edf648f5d4
workflow-basic-steps:2.24
workflow-cps:2648.va9433432b33c
workflow-cps-global-lib:552.vd9cc05b8a2e1
workflow-durable-task-step:1112.vda00e6febcc1
workflow-job:1145.v7f2433caa07f
workflow-multibranch:696.v52535c46f4c9
workflow-scm-step:2.13
workflow-step-api:615.vb09dac339255
workflow-support:804.vba10a18a1476
ws-cleanup:0.40

### What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins randomly deletes slaves before completing job runs.
We could not establish any sort of pattern for when it happens, it is not time dependent as far as I can tell.

We get the following errors:

Caused: java.io.IOException: Unexpected termination of the channel

Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@60d32f79:jenkins-jgqlcq": Remote call on jenkins-jgqlcq failed. The channel is closing down or has closed down

I can see in GCP Log Explorer that Jenkins sends a v1.compute.instances.delete to the node when this happens, however, I cannot find any definition for it.

I set a retention time to 500 just to see that it is not the issue, launch timeout is 300.

### Reproduction steps

1. GCP is spinning up a slave instance to run
2. randomly it will get (or not) a v1.compute.instances.delete from Jenkins.

### Expected Results

All slaves should finish their runs without disconnecting

### Actual Results

Slaves randomly disconnect without any visible pattern

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the GCP Log Explorer entries for v1.compute.instances.delete and the plugin's handling of the 300-second launch timeout and 500-second retention setting. Trace which condition leads to an instance deletion, then verify that an agent can complete its run without an unexpected channel termination or deletion request.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.