jenkinsci / jenkinsci/docker-plugin
[JENKINS-69921] docker plugin agents fail with I/O error and timeout exception
Open
component:docker-plugin
imported-jira-issue
priority:blocker
resolution:unresolved
- Dominant language
- Java
- Stars
- 498
- Forks
- 324
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 2
Description
Container agents fail to shut down and communicate properly with Jenkins master when using the "docker container attach" method for communication. The job runs fine and completes it's task of running a simple sh command, but the agent won't provide monitoring data and fails with a timeout exception.
We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some jobs that won't work with ssh (These jobs use a docker macvlan network to share data over DDS).
We would continue to use JNLP, but it's become very unstable since it went unsupported in Java 11.
Build log
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE
jenkins.log
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
agent console.log
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
---
Originally reported by pjorgen, imported from: docker plugin agents fail with I/O error and timeout exception
Raw content of original issue
Container agents fail to shut down and communicate properly with Jenkins master when using the "docker container attach" method for communication. The job runs fine and completes it's task of running a simple sh command, but the agent won't provide monitoring data and fails with a timeout exception.
We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some jobs that won't work with ssh (These jobs use a docker macvlan network to share data over DDS).
We would continue to use JNLP, but it's become very unstable since it went unsupported in Java 11.
Build log
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILUREjenkins.log
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.agent console.log
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
environment
```
Jenkins 2.356
docker-plugin 1.2.10
docker-java-api-plugin 3.2.13-37.vf3411c9828b9
```
1 attachment
- [Screenshot 2022-10-21 102206.png](https://raw.githubusercontent.com/jenkinsci/attachments-from-jira-issues-misc/refs/heads/main/attachments/59317/Screenshot%202022-10-21%20102206.png)
> 
Contributor guide
Assessment
This issue has not been assessed yet.