jenkinsci / jenkinsci/docker-workflow-plugin

[JENKINS-73561] docker pipeline hangs when container has problems starting

Open
#718 0 comments 0 reactions 0 assignees View on GitHub
component:docker-workflow-plugin imported-jira-issue priority:minor resolution:unresolved
Dominant language
Java
Stars
527
Forks
422
Avg merge
10m
Merged PRs (30d)
1

Description

The following pipeline:

pipeline {

    agent {

        docker {

            image 'quay.io/condaforge/linux-anvil-cos7-x86_64'

        }

    }

    stages {

        stage('Test') {

            steps {

                sh 'echo Hello World'

            }

        }

    }

}

...will run the first time, then all subsequent builds will simply hang at the `sh` step:

[Pipeline] Start of Pipeline

[Pipeline] node

Running on Jenkins in /var/lib/jenkins/workspace/test

[Pipeline] {

[Pipeline] isUnix

[Pipeline] withEnv

[Pipeline] {

[Pipeline] sh

+ docker inspect -f . quay.io/condaforge/linux-anvil-cos7-x86_64

.

[Pipeline] }

[Pipeline] // withEnv

[Pipeline] withDockerContainer

Jenkins does not seem to be running inside a container

$ docker run -t -d -u 115:121 -w /var/lib/jenkins/workspace/test -v /var/lib/jenkins/workspace/test:/var/lib/jenkins/workspace/test:rw,z -v /var/lib/jenkins/workspace/test@​tmp:/var/lib/jenkins/workspace/test@​tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** quay.io/condaforge/linux-anvil-cos7-x86_64 cat

$ docker top 82b5ffd16f2046efca559bb68c910a2230e4f6fedf3bdbf61b6d9e5763ae673b -eo pid,comm

ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).

Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`.

[Pipeline] {

[Pipeline] stage

[Pipeline] { (Test)

[Pipeline] sh

Based on my testing, a full system reboot seems to be the only way to "reset." Removing docker containers and restarting Jenkins did not seem to do the trick on my system. Syslog shows more info: (Note that this might not be the same syslog from the build log above; I've run this pipeline many times to diagnose and the log files are getting hard to keep track of)

Aug  2 18:40:19 networkd-dispatcher[826]: WARNING:Unknown index 5 seen, reloading interface list

Aug  2 18:40:19 systemd-udevd[1763]: Using default interface naming scheme 'v249'.

Aug  2 18:40:19 systemd-udevd[1764]: Using default interface naming scheme 'v249'.

Aug  2 18:40:19 systemd-networkd[807]: veth40bd37e: Link UP

Aug  2 18:40:19 kernel: [  179.207950] docker0: port 1(veth40bd37e) entered blocking state

Aug  2 18:40:19 kernel: [  179.207957] docker0: port 1(veth40bd37e) entered disabled state

Aug  2 18:40:19 kernel: [  179.208044] device veth40bd37e entered promiscuous mode

Aug  2 18:40:20 containerd[902]: time="2024-08-02T18:40:20.502900639Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1

Aug  2 18:40:20 containerd[902]: time="2024-08-02T18:40:20.502958073Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1

Aug  2 18:40:20 containerd[902]: time="2024-08-02T18:40:20.502970759Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

Aug  2 18:40:20 containerd[902]: time="2024-08-02T18:40:20.503572662Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

Aug  2 18:40:20 systemd[1]: Started libcontainer container 7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d.

Aug  2 18:40:20 kernel: [  180.420318] eth0: renamed from veth0dae4b5

Aug  2 18:40:20 systemd-networkd[807]: veth40bd37e: Gained carrier

Aug  2 18:40:20 systemd-networkd[807]: docker0: Gained carrier

Aug  2 18:40:20 kernel: [  180.436377] IPv6: ADDRCONF(NETDEV_CHANGE): veth40bd37e: link becomes ready

Aug  2 18:40:20 kernel: [  180.436419] docker0: port 1(veth40bd37e) entered blocking state

Aug  2 18:40:20 kernel: [  180.436423] docker0: port 1(veth40bd37e) entered forwarding state

Aug  2 18:40:20 kernel: [  180.436467] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready

Aug  2 18:40:21 systemd-networkd[807]: veth40bd37e: Gained IPv6LL

Aug  2 18:40:22 systemd-networkd[807]: docker0: Gained IPv6LL

Aug  2 18:40:26 dockerd[1206]: time="2024-08-02T18:40:26.276620431Z" level=info msg="Container failed to exit within 1s of signal 15 - using the force" container=7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d

Aug  2 18:40:26 systemd[1]: docker-7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d.scope: Deactivated successfully.

Aug  2 18:40:26 dockerd[1206]: time="2024-08-02T18:40:26.488470032Z" level=info msg="ignoring event" container=7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Aug  2 18:40:26 containerd[902]: time="2024-08-02T18:40:26.488395723Z" level=info msg="shim disconnected" id=7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d namespace=moby

Aug  2 18:40:26 containerd[902]: time="2024-08-02T18:40:26.488513952Z" level=warning msg="cleaning up after shim disconnected" id=7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d namespace=moby

Aug  2 18:40:26 containerd[902]: time="2024-08-02T18:40:26.488528019Z" level=info msg="cleaning up dead shim" namespace=moby

Aug  2 18:40:26 systemd-networkd[807]: veth40bd37e: Lost carrier

Aug  2 18:40:26 kernel: [  186.335638] docker0: port 1(veth40bd37e) entered disabled state

Aug  2 18:40:26 kernel: [  186.335823] veth0dae4b5: renamed from eth0

Aug  2 18:40:26 networkd-dispatcher[826]: WARNING:Unknown index 5 seen, reloading interface list

Aug  2 18:40:26 systemd-udevd[1959]: Using default interface naming scheme 'v249'.

Aug  2 18:40:26 systemd-networkd[807]: veth40bd37e: Link DOWN

Aug  2 18:40:26 kernel: [  186.452434] docker0: port 1(veth40bd37e) entered disabled state

Aug  2 18:40:26 kernel: [  186.453120] device veth40bd37e left promiscuous mode

Aug  2 18:40:26 kernel: [  186.453128] docker0: port 1(veth40bd37e) entered disabled state

Aug  2 18:40:26 systemd[1]: run-docker-netns-0d5cf01af203.mount: Deactivated successfully.

Aug  2 18:40:26 systemd[1]: var-lib-docker-overlay2-4f22a701401eb1d93cbe1ffceb4903e84fe6fa305863eaea025f3daeaed537a2-merged.mount: Deactivated successfully.

Aug  2 18:40:27 systemd-networkd[807]: docker0: Lost carrier

Aug  2 18:40:46 kernel: [  206.524406] [NON-UBC BLOCK] IN=enp0s25 OUT= MAC=ff:ff:ff:ff:ff:ff:a4:fc:14:30:b5:b7:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=255 ID=38311 PROTO=UDP SPT=68 DPT=67 LEN=308 

Aug  2 18:40:47 kernel: [  207.648367] [NON-UBC BLOCK] IN=enp0s25 OUT= MAC=ff:ff:ff:ff:ff:ff:a4:fc:14:30:b5:b7:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=255 ID=38312 PROTO=UDP SPT=68 DPT=67 LEN=308 

Aug  2 18:40:52 systemd[1]: var-lib-docker-overlay2-19e12f681843e54a84fcc10229c768334f2f4a214828ef13108eb45e8057671c\x2dinit-merged.mount: Deactivated successfully.

Aug  2 18:40:53 systemd[1]: var-lib-docker-overlay2-19e12f681843e54a84fcc10229c768334f2f4a214828ef13108eb45e8057671c-merged.mount: Deactivated successfully.

Aug  2 18:40:53 systemd-udevd[2034]: Using default interface naming scheme 'v249'.

Aug  2 18:40:53 systemd-udevd[2035]: Using default interface naming scheme 'v249'.

Aug  2 18:40:53 networkd-dispatcher[826]: WARNING:Unknown index 7 seen, reloading interface list

Aug  2 18:40:53 kernel: [  213.510757] docker0: port 1(veth5bea384) entered blocking state

Aug  2 18:40:53 kernel: [  213.510764] docker0: port 1(veth5bea384) entered disabled state

Aug  2 18:40:53 systemd-networkd[807]: veth5bea384: Link UP

Aug  2 18:40:53 kernel: [  213.511917] device veth5bea384 entered promiscuous mode

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.148781255Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.148834946Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.148848600Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.148972580Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

Aug  2 18:40:54 systemd[1]: Started libcontainer container 0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776.

Aug  2 18:40:54 kernel: [  213.983562] eth0: renamed from vetha27c4b5

Aug  2 18:40:54 systemd-networkd[807]: veth5bea384: Gained carrier

Aug  2 18:40:54 systemd-networkd[807]: docker0: Gained carrier

Aug  2 18:40:54 kernel: [  214.003356] IPv6: ADDRCONF(NETDEV_CHANGE): veth5bea384: link becomes ready

Aug  2 18:40:54 kernel: [  214.003395] docker0: port 1(veth5bea384) entered blocking state

Aug  2 18:40:54 kernel: [  214.003399] docker0: port 1(veth5bea384) entered forwarding state

Aug  2 18:40:54 systemd[1]: docker-0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776.scope: Deactivated successfully.

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.878260626Z" level=info msg="shim disconnected" id=0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776 namespace=moby

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.878330353Z" level=warning msg="cleaning up after shim disconnected" id=0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776 namespace=moby

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.878349162Z" level=info msg="cleaning up dead shim" namespace=moby

Aug  2 18:40:54 dockerd[1206]: time="2024-08-02T18:40:54.878307520Z" level=info msg="ignoring event" container=0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Aug  2 18:40:55 systemd-networkd[807]: veth5bea384: Lost carrier

Aug  2 18:40:55 kernel: [  214.779301] docker0: port 1(veth5bea384) entered disabled state

Aug  2 18:40:55 kernel: [  214.779561] vetha27c4b5: renamed from eth0

Aug  2 18:40:55 networkd-dispatcher[826]: WARNING:Unknown index 7 seen, reloading interface list

Aug  2 18:40:55 systemd-udevd[2048]: Using default interface naming scheme 'v249'.

Aug  2 18:40:55 systemd-networkd[807]: veth5bea384: Link DOWN

Aug  2 18:40:55 kernel: [  214.958992] docker0: port 1(veth5bea384) entered disabled state

Aug  2 18:40:55 kernel: [  214.959835] device veth5bea384 left promiscuous mode

Aug  2 18:40:55 kernel: [  214.959843] docker0: port 1(veth5bea384) entered disabled state

Aug  2 18:40:55 systemd[1]: run-docker-netns-e80d8c1be79b.mount: Deactivated successfully.

Aug  2 18:40:55 systemd[1]: var-lib-docker-overlay2-19e12f681843e54a84fcc10229c768334f2f4a214828ef13108eb45e8057671c-merged.mount: Deactivated successfully.

Aug  2 18:40:56 systemd-networkd[807]: docker0: Lost carrier

Aug  2 18:40:56 dockerd[1206]: time="2024-08-02T18:40:56.307380517Z" level=error msg="Error setting up exec command in container 0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776: Container 0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776 is not running"

The last line is key; The container isn't running properly. `docker logs` shows why:

/usr/bin/id: cannot find name for user ID 115

bash: unalias: cp: not found

useradd: Permission denied.

useradd: cannot lock /etc/passwd; try again later.

chown: invalid user: 'conda:conda'

cp: cannot create directory '/home/conda': Permission denied

rm: cannot remove '/opt/conda/.condarc': Permission denied

cp: cannot stat '/root/.condarc': Permission denied

bash: cd: /home/conda: No such file or directory

su-exec: setgroups(121): Operation not permitted

This appears to be a failed execution of this script in the conda forge image. In short, the conda forge image is being rather nonstandard and dynamically creating the user for container execution to try and match the host UID. Hence, the bulk of this issue is nonstandard Docker practices in the image. I think the issues in Jenkins are limited to:



  1. The build does not repeat the `Container is not running` message, and appears to have no record of the nonzero exit code of the corresponding command. The build really should fail at this step rather than hanging.

  2. Behavior is inconsistent between execution steps; If a container is doing nonstandard behavior, an error should only pop up in one build, and subsequent builds should show the same behavior (rather than working once, then subsequent steps failing).

The only workaround is to invoke Docker commands manually. I'm labeling this bug as minor because it appears limited to certain containers that will not work in Jenkins anyway.

---
Originally reported by kb1rd, imported from: docker pipeline hangs when container has problems starting


  • status: Open
  • priority: Minor
  • component(s): docker-workflow-plugin
  • resolution: Unresolved
  • votes: 0
  • watchers: 1
  • imported: 2025-12-07

Raw content of original issue

The following pipeline:




pipeline {

    agent {

        docker {

            image 'quay.io/condaforge/linux-anvil-cos7-x86_64'

        }

    }

    stages {

        stage('Test') {

            steps {

                sh 'echo Hello World'

            }

        }

    }

}




...will run the first time, then all subsequent builds will simply hang at the `sh` step:




[Pipeline] Start of Pipeline

[Pipeline] node

Running on Jenkins in /var/lib/jenkins/workspace/test

[Pipeline] {

[Pipeline] isUnix

[Pipeline] withEnv

[Pipeline] {

[Pipeline] sh

+ docker inspect -f . quay.io/condaforge/linux-anvil-cos7-x86_64

.

[Pipeline] }

[Pipeline] // withEnv

[Pipeline] withDockerContainer

Jenkins does not seem to be running inside a container

$ docker run -t -d -u 115:121 -w /var/lib/jenkins/workspace/test -v /var/lib/jenkins/workspace/test:/var/lib/jenkins/workspace/test:rw,z -v /var/lib/jenkins/workspace/test@tmp:/var/lib/jenkins/workspace/test@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** quay.io/condaforge/linux-anvil-cos7-x86_64 cat

$ docker top 82b5ffd16f2046efca559bb68c910a2230e4f6fedf3bdbf61b6d9e5763ae673b -eo pid,comm

ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).

Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`.

[Pipeline] {

[Pipeline] stage

[Pipeline] { (Test)

[Pipeline] sh




Based on my testing, a full system reboot seems to be the only way to "reset." Removing docker containers and restarting Jenkins did not seem to do the trick on my system. Syslog shows more info: (Note that this might not be the same syslog from the build log above; I've run this pipeline many times to diagnose and the log files are getting hard to keep track of)




Aug  2 18:40:19 networkd-dispatcher[826]: WARNING:Unknown index 5 seen, reloading interface list

Aug  2 18:40:19 systemd-udevd[1763]: Using default interface naming scheme 'v249'.

Aug  2 18:40:19 systemd-udevd[1764]: Using default interface naming scheme 'v249'.

Aug  2 18:40:19 systemd-networkd[807]: veth40bd37e: Link UP

Aug  2 18:40:19 kernel: [  179.207950] docker0: port 1(veth40bd37e) entered blocking state

Aug  2 18:40:19 kernel: [  179.207957] docker0: port 1(veth40bd37e) entered disabled state

Aug  2 18:40:19 kernel: [  179.208044] device veth40bd37e entered promiscuous mode

Aug  2 18:40:20 containerd[902]: time="2024-08-02T18:40:20.502900639Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1

Aug  2 18:40:20 containerd[902]: time="2024-08-02T18:40:20.502958073Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1

Aug  2 18:40:20 containerd[902]: time="2024-08-02T18:40:20.502970759Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

Aug  2 18:40:20 containerd[902]: time="2024-08-02T18:40:20.503572662Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

Aug  2 18:40:20 systemd[1]: Started libcontainer container 7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d.

Aug  2 18:40:20 kernel: [  180.420318] eth0: renamed from veth0dae4b5

Aug  2 18:40:20 systemd-networkd[807]: veth40bd37e: Gained carrier

Aug  2 18:40:20 systemd-networkd[807]: docker0: Gained carrier

Aug  2 18:40:20 kernel: [  180.436377] IPv6: ADDRCONF(NETDEV_CHANGE): veth40bd37e: link becomes ready

Aug  2 18:40:20 kernel: [  180.436419] docker0: port 1(veth40bd37e) entered blocking state

Aug  2 18:40:20 kernel: [  180.436423] docker0: port 1(veth40bd37e) entered forwarding state

Aug  2 18:40:20 kernel: [  180.436467] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready

Aug  2 18:40:21 systemd-networkd[807]: veth40bd37e: Gained IPv6LL

Aug  2 18:40:22 systemd-networkd[807]: docker0: Gained IPv6LL

Aug  2 18:40:26 dockerd[1206]: time="2024-08-02T18:40:26.276620431Z" level=info msg="Container failed to exit within 1s of signal 15 - using the force" container=7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d

Aug  2 18:40:26 systemd[1]: docker-7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d.scope: Deactivated successfully.

Aug  2 18:40:26 dockerd[1206]: time="2024-08-02T18:40:26.488470032Z" level=info msg="ignoring event" container=7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Aug  2 18:40:26 containerd[902]: time="2024-08-02T18:40:26.488395723Z" level=info msg="shim disconnected" id=7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d namespace=moby

Aug  2 18:40:26 containerd[902]: time="2024-08-02T18:40:26.488513952Z" level=warning msg="cleaning up after shim disconnected" id=7ede9e443ad022290032082637ded80263a23c6764de6dbe49143577c0cd8a5d namespace=moby

Aug  2 18:40:26 containerd[902]: time="2024-08-02T18:40:26.488528019Z" level=info msg="cleaning up dead shim" namespace=moby

Aug  2 18:40:26 systemd-networkd[807]: veth40bd37e: Lost carrier

Aug  2 18:40:26 kernel: [  186.335638] docker0: port 1(veth40bd37e) entered disabled state

Aug  2 18:40:26 kernel: [  186.335823] veth0dae4b5: renamed from eth0

Aug  2 18:40:26 networkd-dispatcher[826]: WARNING:Unknown index 5 seen, reloading interface list

Aug  2 18:40:26 systemd-udevd[1959]: Using default interface naming scheme 'v249'.

Aug  2 18:40:26 systemd-networkd[807]: veth40bd37e: Link DOWN

Aug  2 18:40:26 kernel: [  186.452434] docker0: port 1(veth40bd37e) entered disabled state

Aug  2 18:40:26 kernel: [  186.453120] device veth40bd37e left promiscuous mode

Aug  2 18:40:26 kernel: [  186.453128] docker0: port 1(veth40bd37e) entered disabled state

Aug  2 18:40:26 systemd[1]: run-docker-netns-0d5cf01af203.mount: Deactivated successfully.

Aug  2 18:40:26 systemd[1]: var-lib-docker-overlay2-4f22a701401eb1d93cbe1ffceb4903e84fe6fa305863eaea025f3daeaed537a2-merged.mount: Deactivated successfully.

Aug  2 18:40:27 systemd-networkd[807]: docker0: Lost carrier

Aug  2 18:40:46 kernel: [  206.524406] [NON-UBC BLOCK] IN=enp0s25 OUT= MAC=ff:ff:ff:ff:ff:ff:a4:fc:14:30:b5:b7:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=255 ID=38311 PROTO=UDP SPT=68 DPT=67 LEN=308 

Aug  2 18:40:47 kernel: [  207.648367] [NON-UBC BLOCK] IN=enp0s25 OUT= MAC=ff:ff:ff:ff:ff:ff:a4:fc:14:30:b5:b7:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=255 ID=38312 PROTO=UDP SPT=68 DPT=67 LEN=308 

Aug  2 18:40:52 systemd[1]: var-lib-docker-overlay2-19e12f681843e54a84fcc10229c768334f2f4a214828ef13108eb45e8057671c\x2dinit-merged.mount: Deactivated successfully.

Aug  2 18:40:53 systemd[1]: var-lib-docker-overlay2-19e12f681843e54a84fcc10229c768334f2f4a214828ef13108eb45e8057671c-merged.mount: Deactivated successfully.

Aug  2 18:40:53 systemd-udevd[2034]: Using default interface naming scheme 'v249'.

Aug  2 18:40:53 systemd-udevd[2035]: Using default interface naming scheme 'v249'.

Aug  2 18:40:53 networkd-dispatcher[826]: WARNING:Unknown index 7 seen, reloading interface list

Aug  2 18:40:53 kernel: [  213.510757] docker0: port 1(veth5bea384) entered blocking state

Aug  2 18:40:53 kernel: [  213.510764] docker0: port 1(veth5bea384) entered disabled state

Aug  2 18:40:53 systemd-networkd[807]: veth5bea384: Link UP

Aug  2 18:40:53 kernel: [  213.511917] device veth5bea384 entered promiscuous mode

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.148781255Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.148834946Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.148848600Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.148972580Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

Aug  2 18:40:54 systemd[1]: Started libcontainer container 0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776.

Aug  2 18:40:54 kernel: [  213.983562] eth0: renamed from vetha27c4b5

Aug  2 18:40:54 systemd-networkd[807]: veth5bea384: Gained carrier

Aug  2 18:40:54 systemd-networkd[807]: docker0: Gained carrier

Aug  2 18:40:54 kernel: [  214.003356] IPv6: ADDRCONF(NETDEV_CHANGE): veth5bea384: link becomes ready

Aug  2 18:40:54 kernel: [  214.003395] docker0: port 1(veth5bea384) entered blocking state

Aug  2 18:40:54 kernel: [  214.003399] docker0: port 1(veth5bea384) entered forwarding state

Aug  2 18:40:54 systemd[1]: docker-0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776.scope: Deactivated successfully.

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.878260626Z" level=info msg="shim disconnected" id=0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776 namespace=moby

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.878330353Z" level=warning msg="cleaning up after shim disconnected" id=0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776 namespace=moby

Aug  2 18:40:54 containerd[902]: time="2024-08-02T18:40:54.878349162Z" level=info msg="cleaning up dead shim" namespace=moby

Aug  2 18:40:54 dockerd[1206]: time="2024-08-02T18:40:54.878307520Z" level=info msg="ignoring event" container=0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Aug  2 18:40:55 systemd-networkd[807]: veth5bea384: Lost carrier

Aug  2 18:40:55 kernel: [  214.779301] docker0: port 1(veth5bea384) entered disabled state

Aug  2 18:40:55 kernel: [  214.779561] vetha27c4b5: renamed from eth0

Aug  2 18:40:55 networkd-dispatcher[826]: WARNING:Unknown index 7 seen, reloading interface list

Aug  2 18:40:55 systemd-udevd[2048]: Using default interface naming scheme 'v249'.

Aug  2 18:40:55 systemd-networkd[807]: veth5bea384: Link DOWN

Aug  2 18:40:55 kernel: [  214.958992] docker0: port 1(veth5bea384) entered disabled state

Aug  2 18:40:55 kernel: [  214.959835] device veth5bea384 left promiscuous mode

Aug  2 18:40:55 kernel: [  214.959843] docker0: port 1(veth5bea384) entered disabled state

Aug  2 18:40:55 systemd[1]: run-docker-netns-e80d8c1be79b.mount: Deactivated successfully.

Aug  2 18:40:55 systemd[1]: var-lib-docker-overlay2-19e12f681843e54a84fcc10229c768334f2f4a214828ef13108eb45e8057671c-merged.mount: Deactivated successfully.

Aug  2 18:40:56 systemd-networkd[807]: docker0: Lost carrier

Aug  2 18:40:56 dockerd[1206]: time="2024-08-02T18:40:56.307380517Z" level=error msg="Error setting up exec command in container 0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776: Container 0d092ae3db416822aab25da076b02c5ebbc2a6453c10cd23760b29c1f6d69776 is not running"




The last line is key; The container isn't running properly. `docker logs` shows why:




/usr/bin/id: cannot find name for user ID 115

bash: unalias: cp: not found

useradd: Permission denied.

useradd: cannot lock /etc/passwd; try again later.

chown: invalid user: 'conda:conda'

cp: cannot create directory '/home/conda': Permission denied

rm: cannot remove '/opt/conda/.condarc': Permission denied

cp: cannot stat '/root/.condarc': Permission denied

bash: cd: /home/conda: No such file or directory

su-exec: setgroups(121): Operation not permitted




This appears to be a failed execution of this script in the conda forge image. In short, the conda forge image is being rather nonstandard and dynamically creating the user for container execution to try and match the host UID. Hence, the bulk of this issue is nonstandard Docker practices in the image. I think the issues in Jenkins are limited to:


    

    	
  1. The build does not repeat the `Container is not running` message, and appears to have no record of the nonzero exit code of the corresponding command. The build really should fail at this step rather than hanging.
    2. 

    	
  2. Behavior is inconsistent between execution steps; If a container is doing nonstandard behavior, an error should only pop up in one build, and subsequent builds should show the same behavior (rather than working once, then subsequent steps failing).
    3. 




The only workaround is to invoke Docker commands manually. I'm labeling this bug as minor because it appears limited to certain containers that will not work in Jenkins anyway.

environment

```
Linux 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Jenkins and jdk 17 are installed via apt on Ubuntu Server 22.04.

Plugin Versions:

Apache HttpComponents Client 4.x API Plugin 4.5.14-208.v438351942757 true

Apache HttpComponents Client 5.x API Plugin 5.3.1-110.v77252fb_d4da_5 true

ASM API Plugin 9.7-33.v4d23ef79fcc8 true

Authentication Tokens API Plugin 1.119.v50285141b_7e1 true

Bootstrap 5 API Plugin 5.3.3-1 true

bouncycastle API Plugin 2.30.1.78.1-248.ve27176eb_46cb_ true

Branch API Plugin 2.1178.v969d9eb_c728e true

Build Name and Description Setter 2.4.3 true

Build Timeout 1.33 true

Caffeine API Plugin 3.1.8-133.v17b_1ff2e0599 true

Checks API plugin 2.2.0 true

Cloud Statistics Plugin 336.v788e4055508b_ true

commons-lang3 v3.x Jenkins API Plugin 3.14.0-76.vda_5591261cfe true

commons-text API Plugin 1.12.0-129.v99a_50df237f7 true

Conditional BuildStep 1.4.3 true

Config File Provider Plugin 973.vb_a_80ecb_9a_4d0 true

Configuration as Code Plugin 1836.vccda_4a_122a_a_e true

Copy Artifact Plugin 749.vfb_dca_a_9b_6549 true

Credentials Binding Plugin 681.vf91669a_32e45 true

Credentials Plugin 1371.vfee6b_095f0a_3 true

Dark Theme 439.vdef09f81f85e true

Dashboard View 2.508.va_74654f026d1 true

Display URL API 2.204.vf6fddd8a_8b_e9 true

Docker API Plugin 3.3.6-90.ve7c5c7535ddd true

Docker Commons Plugin 439.va_3cb_0a_6a_fb_29 true

Docker Pipeline 580.vc0c340686b_54 true

Docker plugin 1.6.2 true

Durable Task Plugin 555.v6802fe0f0b_82 true

ECharts API Plugin 5.5.0-1 true

EDDSA API Plugin 0.3.0-4.v84c6f0f4969e true

Embeddable Build Status Plugin 487.va_0ef04c898a_2 true

Folders Plugin 6.928.v7c780211d66e true

Font Awesome API Plugin 6.5.2-1 true

Git client plugin 5.0.0 true

Git plugin 5.2.2 true

GitHub API Plugin 1.321-468.v6a_9f5f2d5a_7e true

GitHub Branch Source Plugin 1793.v1831e9c68d77 true

GitHub plugin 1.39.0 true

Gson API Plugin 2.11.0-41.v019fcf6125dc true

Instance Identity 185.v303dc7c645f9 true

Ionicons API 74.v93d5eb_813d5f true

Jackson 2 API Plugin 2.17.0-379.v02de8ec9f64c true

Jakarta Activation API 2.1.3-1 true

Jakarta Mail API 2.1.3-1 true

Java JSON Web Token (JJWT) Plugin 0.11.5-112.ve82dfb_224b_a_d true

JavaBeans Activation Framework (JAF) API 1.2.0-7 true

JAXB plugin 2.3.9-1 true

Joda Time API Plugin 2.12.7-29.v5a_b_e3a_82269a_ true

JQuery3 API Plugin 3.7.1-2 true

JSON Api Plugin 20240303-41.v94e11e6de726 true

JSON Path API Plugin 2.9.0-58.v62e3e85b_a_655 true

JUnit Plugin 1280.v310a_78b_9a_1e0 true

Mailer Plugin 472.vf7c289a_4b_420 true

Matrix Authorization Strategy Plugin 3.2.2 true

Matrix Project Plugin 832.va_66e270d2946 true

Metrics Plugin 4.2.21-451.vd51df8df52ec true

Mina SSHD API :: Common 2.13.1-117.v2f1a_b_66ff91d true

Mina SSHD API :: Core 2.13.1-117.v2f1a_b_66ff91d true

OkHttp Plugin 4.11.0-172.vda_da_1feeb_c6e true

OWASP Markup Formatter Plugin 162.v0e6ec0fcfcf6 true

PAM Authentication plugin 1.11 true

Parameterized Trigger plugin 806.vf6fff3e28c3e true

Pipeline 600.vb_57cdd26fdd7 true

Pipeline Graph Analysis Plugin 216.vfd8b_ece330ca_ true

Pipeline Graph View Plugin 313.v1322ce83d680 true

Pipeline: API 1322.v857eeeea_9902 true

Pipeline: Basic Steps 1058.vcb_fc1e3a_21a_9 true

Pipeline: Build Step 540.vb_e8849e1a_b_d8 true

Pipeline: Declarative 2.2205.vc9522a_9d5711 true

Pipeline: Declarative Extension Points API 2.2205.vc9522a_9d5711 true

Pipeline: GitHub Groovy Libraries 61.v629f2cc41d83 true

Pipeline: Groovy 3922.va_f73b_7c4246b_ true

Pipeline: Groovy Libraries 727.ve832a_9244dfa_ true

Pipeline: Input Step 495.ve9c153f6067b_ true

Pipeline: Job 1400.v7fd111b_ec82f true

Pipeline: Milestone Step 119.vdfdc43fc3b_9a_ true

Pipeline: Model API 2.2205.vc9522a_9d5711 true

Pipeline: Multibranch 783.787.v50539468395f true

Pipeline: Nodes and Processes 1364.v2fd76fb_6fd41 true

Pipeline: SCM Step 427.v4ca_6512e7df1 true

Pipeline: Stage Step 312.v8cd10304c27a_ true

Pipeline: Stage Tags Metadata 2.2205.vc9522a_9d5711 true

Pipeline: Step API 678.v3ee58b_469476 true

Pipeline: Supporting APIs 920.v59f71ce16f04 true

Plain Credentials Plugin 183.va_de8f1dd5a_2b_ true

Plugin Utilities API Plugin 4.1.0 true

Prism API Plugin 1.29.0-15 true

Resource Disposer Plugin 0.23 true

Run Condition Plugin 1.7 true

SCM API Plugin 696.v778d637b_a_762 true

Script Security Plugin 1341.va_2819b_414686 true

SnakeYAML API Plugin 2.2-111.vc6598e30cc65 true

SSH Agent Plugin 376.v8933585c69d3 true

SSH Build Agents plugin 2.973.v0fa_8c0dea_f9f true

SSH Credentials Plugin 343.v884f71d78167 true

Structs Plugin 338.v848422169819 true

Theme Manager 262.vc57ee4a_eda_5d true

Throttle Concurrent Builds Plug-in 2.14 true

Timestamper 1.27 true

Token Macro Plugin 400.v35420b_922dcb_ true

Trilead API Plugin 2.147.vb_73cc728a_32e true

Variant Plugin 60.v7290fc0eb_b_cd true

Workspace Cleanup Plugin 0.46 true
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.