[JENKINS-63253] pipeline + docker + dir() causes: process apparently never started in ...

So there's this weird behavior when using the built-in dir() method with a pipeline, running on a docker container and it causes anything in the dir block to hang. Please note the example below, as it explains it better than I can describe it:

pipeline {
    agent {
docker {
    label '********'
    image '**********'
    registryUrl '************'
    registryCredentialsId '*******'
    args '--user root:root'
}
    }
    stages {
stage('dir-testing') {
    stages {
stage('without dir') {
    steps {
sh 'cd /opt && ls -l'
    }
}
stage('with dir') {
    steps {
dir('/opt') {
    sh 'ls -l'
}
    }
}
    }
    post {
always {
    cleanWs()
}
    }
}
    }
}

And the result is:

Started by user **********
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on ************ in /var/jenkins/workspace/test-cwd-bug
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withDockerRegistry
Using the existing docker config file.Removing blacklisted property: auths$ docker login -u ******** -p ******** *********
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . **********

Error: No such object: ***********
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . ****************
.
[Pipeline] withDockerContainer
************* does not seem to be running inside a container
$ docker run -t -d -u 0:0 --user root:root -w /var/jenkins/workspace/test-cwd-bug -v /var/jenkins/workspace/test-cwd-bug:/var/jenkins/workspace/test-cwd-bug:rw,z -v /var/jenkins/workspace/test-cwd-bug@​tmp:/var/jenkins/workspace/test-cwd-bug@​tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** ********************* cat
$ docker top 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (dir-testing)
[Pipeline] stage
[Pipeline] { (without dir)
[Pipeline] sh
+ cd /opt
+ ls -l
total 8
drwxr-xr-x 4 root root 4096 Jul 24 15:29 artifactory-scripts
drwxr-xr-x 1  608  500 4096 Jul 24 15:39 cv25_linux_sdk_2.5
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (with dir)
[Pipeline] dir
Running in /opt
[Pipeline] {
[Pipeline] sh
process apparently never started in /opt@​tmp/durable-5a20a76a
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
Post stage
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16
$ docker rm -f 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE

So there's this weird behavior when using the built-in dir() method with a pipeline, running on a docker container and it causes anything in the dir block to hang. Please note the example below, as it explains it better than I can describe it:

 

pipeline {
    agent {
        docker {
            label '********'
            image '**********'
            registryUrl '************'
            registryCredentialsId '*******'
            args '--user root:root'
        }
    }
    stages {
        stage('dir-testing') {
            stages {
                stage('without dir') {
                    steps {
                        sh 'cd /opt && ls -l'
                    }
                }
                stage('with dir') {
                    steps {
                        dir('/opt') {
                            sh 'ls -l'
                        }
                    }
                }
            }
            post {
                always {
                    cleanWs()
                }
            }
        }
    }
}


And the result is:

Started by user **********
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on ************ in /var/jenkins/workspace/test-cwd-bug
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withDockerRegistry
Using the existing docker config file.Removing blacklisted property: auths$ docker login -u ******** -p ******** *********
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . **********
Error: No such object: ***********

[Pipeline] isUnix

[Pipeline] sh

docker inspect -f . ****************

.

[Pipeline] withDockerContainer

************* does not seem to be running inside a container

$ docker run -t -d -u 0:0 --user root:root -w /var/jenkins/workspace/test-cwd-bug -v /var/jenkins/workspace/test-cwd-bug:/var/jenkins/workspace/test-cwd-bug:rw,z -v /var/jenkins/workspace/test-cwd-bug@tmp:/var/jenkins/workspace/test-cwd-bug@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** ********************* cat

$ docker top 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16 -eo pid,comm

[Pipeline] {

[Pipeline] stage

[Pipeline] { (dir-testing)

[Pipeline] stage

[Pipeline] { (without dir)

[Pipeline] sh
cd /opt
ls -l

total 8

drwxr-xr-x 4 root root 4096 Jul 24 15:29 artifactory-scripts

drwxr-xr-x 1  608  500 4096 Jul 24 15:39 cv25_linux_sdk_2.5

[Pipeline] }

[Pipeline] // stage

[Pipeline] stage

[Pipeline] { (with dir)

[Pipeline] dir

Running in /opt

[Pipeline] {

[Pipeline] sh

process apparently never started in /opt@tmp/durable-5a20a76a

(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

[Pipeline] }

[Pipeline] // dir

[Pipeline] }

[Pipeline] // stage

Post stage

[Pipeline] cleanWs

[WS-CLEANUP] Deleting project workspace...

[WS-CLEANUP] Deferred wipeout is used...

[WS-CLEANUP] done

[Pipeline] }

[Pipeline] // stage

[Pipeline] }

$ docker stop --time=1 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16

$ docker rm -f 969a08a99a24c314d5d80f2cbf77920db4e269524d1af6738c0ddc5417da3f16

[Pipeline] // withDockerContainer

[Pipeline] }

[Pipeline] // withDockerRegistry

[Pipeline] }

[Pipeline] // withEnv

[Pipeline] }

[Pipeline] // node

[Pipeline] End of Pipeline

ERROR: script returned exit code -2

Finished: FAILURE



 

OS Jenkins Master/Slave: Ubuntu x64 16.04<br/>
Jenkins server version: 2.235.2<br/>
durable task plugin version: 1.34<br/>
all other plugins: latest version (for 2020-07-29)

Reproduziere die bereitgestellte Pipeline mit dem docker agent, dem verschachtelten dir('/opt') und sh-Schritten und aktiviere anschließend die erwähnte Einstellung BourneShellScript.LAUNCH_DIAGNOSTICS. Vergleiche den funktionierenden Befehl außerhalb von dir mit dem fehlschlagenden Befehl innerhalb davon; abgeschlossen bedeutet, dass der dir-Block seinen Prozess erfolgreich startet und die Pipeline ohne den durable-task-Fehler abgeschlossen wird.