[JENKINS-43599] Distributed locks occasionally failing across nodes

Are pipeline locks expected to behave when used by two nodes simultaneously?

My Jenkinsfile defines the following helper -

def log(msg) {

  echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)

}


def lockDB(number, callback) {

  try {

    log "${number}: Acquiring lock ..."

    lock(resource: "boom-test-database-${number}", inversePrecedence: true) {

      log "${number}: Acquired!"

      callback()

      log "${number}: Releasing..."

    }

  } finally {

    log "${number}: Released!"

  }

}

I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.

[featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA

[featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...

[Pipeline] [featureTests] lock

[featureTests] Trying to acquire lock on [boom-test-database-1]

[featureTests] Found 0 available resource(s). Waiting for correct amount: 1.

[featureTests] [boom-test-database-1] is locked, waiting...

[featureTests] Lock acquired on [boom-test-database-1]

[featureTests] 2017-04-07T14:45:57: 1: Acquired!

... running tests

[featureTests] Lock released on resource [boom-test-database-1]

[featureTests] 2017-04-07T14:52:04: 1: Released!

[featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ

[featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...

[Pipeline] [featureTests] lock

[featureTests] Trying to acquire lock on [boom-test-database-1]

[featureTests] Found 0 available resource(s). Waiting for correct amount: 1.

[featureTests] [boom-test-database-1] is locked, waiting...

[featureTests] Lock acquired on [boom-test-database-1]

[featureTests] 2017-04-07T14:51:24: 1: Acquired!

... running tests

[featureTests] Lock released on resource [boom-test-database-1]

[featureTests] 2017-04-07T14:56:43: 1: Released!

Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?

---
Originally reported by jdelstrother, imported from: Distributed locks occasionally failing across nodes

• status: Open
  
• priority: Minor
  
• component(s): lockable-resources-plugin
  
• label(s): lock
  
• resolution: Unresolved
  
• votes: 0
  
• watchers: 4
  
• imported: 20251217-081411
  

Are pipeline locks expected to behave when used by two nodes simultaneously?


My Jenkinsfile defines the following helper - 


Jenkinsfile


def log(msg) {

  echo((new Date().format("yyyy-MM-dd'T'HH:mm:ss: ")) + msg)

}


def lockDB(number, callback) {

  try {

    log "${number}: Acquiring lock ..."

    lock(resource: "boom-test-database-${number}", inversePrecedence: true) {

      log "${number}: Acquired!"

      callback()

      log "${number}: Releasing..."

    }

  } finally {

    log "${number}: Released!"

  }

}





I somewhat frequently see two nodes manage to acquire the same lock simultaneously, as in the following example, where master locked a resource from 14:45:57 -> 14:52:04, and the slave locked the same resource from 14:51:24 -> 14:56:43.


Job1


[featureTests] Running on master in /var/jenkins_home/workspace/BoomMultiPipeline_master-YRHUUT2RENAD5FNHS4YJ5GQKTJDN2M7B3KGAU3ELBWX7MZ2AEKBA

[featureTests] 2017-04-07T14:40:44: 1: Acquiring lock ...

[Pipeline] [featureTests] lock

[featureTests] Trying to acquire lock on [boom-test-database-1]

[featureTests] Found 0 available resource(s). Waiting for correct amount: 1.

[featureTests] [boom-test-database-1] is locked, waiting...

[featureTests] Lock acquired on [boom-test-database-1]

[featureTests] 2017-04-07T14:45:57: 1: Acquired!

... running tests

[featureTests] Lock released on resource [boom-test-database-1]

[featureTests] 2017-04-07T14:52:04: 1: Released!





Job2


[featureTests] Running on slave in /var/jenkins_home/workspace/eline_tb_remaining_indices2-DTE7IMSYRVCF6GO3T5IRRR52R4YQ3GSRFNEEKCRH7FGSUBQB4SFQ

[featureTests] 2017-04-07T14:49:33: 1: Acquiring lock ...

[Pipeline] [featureTests] lock

[featureTests] Trying to acquire lock on [boom-test-database-1]

[featureTests] Found 0 available resource(s). Waiting for correct amount: 1.

[featureTests] [boom-test-database-1] is locked, waiting...

[featureTests] Lock acquired on [boom-test-database-1]

[featureTests] 2017-04-07T14:51:24: 1: Acquired!

... running tests

[featureTests] Lock released on resource [boom-test-database-1]

[featureTests] 2017-04-07T14:56:43: 1: Released!





Is this a bug, or am I misunderstanding how locking is supposed to work in Jenkins?

environment

```
System Properties

Name ↓

Value

awt.toolkit sun.awt.X11.XToolkit

executable-war /usr/share/jenkins/jenkins.war

file.encoding UTF-8

file.encoding.pkg sun.io

file.separator /

java.awt.graphicsenv sun.awt.X11GraphicsEnvironment

java.awt.headless true

java.awt.printerjob sun.print.PSPrinterJob

java.class.path /usr/share/jenkins/jenkins.war

java.class.version 52.0

java.endorsed.dirs /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/endorsed

java.ext.dirs /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext

java.home /usr/lib/jvm/java-8-openjdk-amd64/jre

java.io.tmpdir /tmp

java.library.path /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib

java.net.preferIPv4Stack true

java.runtime.name OpenJDK Runtime Environment

java.runtime.version 1.8.0_121-8u121-b13-1~bpo8+1-b13

java.specification.name Java Platform API Specification

java.specification.vendor Oracle Corporation

java.specification.version 1.8

java.vendor Oracle Corporation

java.vendor.url http://java.oracle.com/

java.vendor.url.bug http://bugreport.sun.com/bugreport/

java.version 1.8.0_121

java.vm.info mixed mode

java.vm.name OpenJDK 64-Bit Server VM

java.vm.specification.name Java Virtual Machine Specification

java.vm.specification.vendor Oracle Corporation

java.vm.specification.version 1.8

java.vm.vendor Oracle Corporation

java.vm.version 25.121-b13

jenkins.install.runSetupWizard false

jna.loaded true

jna.platform.library.path /usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/lib64:/usr/lib:/lib:/usr/local/lib

jnidispatch.path /tmp/jna--1712433994/jna2823813371334379938.tmp

line.separator

mail.smtp.sendpartial true

mail.smtps.sendpartial true

os.arch amd64

os.name Linux

os.version 4.4.41-35.53.amzn1.x86_64

path.separator :

sun.arch.data.model 64

sun.boot.class.path /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes

sun.boot.library.path /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64

sun.cpu.endian little

sun.cpu.isalist

sun.font.fontmanager sun.awt.X11FontManager

sun.io.unicode.encoding UnicodeLittle

sun.java.command /usr/share/jenkins/jenkins.war

sun.java.launcher SUN_STANDARD

sun.jnu.encoding UTF-8

sun.management.compiler HotSpot 64-Bit Tiered Compilers

sun.os.patch.level unknown

svnkit.http.methods Digest,Basic,NTLM,Negotiate

svnkit.ssh2.persistent false

user.dir /var/jenkins_home

user.home /var/jenkins_home

user.language en

user.name jenkins

user.timezone Etc/UTC

Environment Variables

Name ↓

Value

HOME /var/jenkins_home

HOSTNAME 576c44275689

JAVA_DEBIAN_VERSION 8u121-b13-1~bpo8+1

JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

JAVA_OPTS -Djenkins.install.runSetupWizard=false

JAVA_VERSION 8u121

JENKINS_HOME /var/jenkins_home

JENKINS_SLAVE_AGENT_PORT 50000

JENKINS_UC https://updates.jenkins.io

JENKINS_VERSION 2.45

LANG C.UTF-8

MAVEN_HOME /usr/share/maven

MAVEN_VERSION 3.3.9

TINI_SHA afbf8de8a63ce8e4f18cb3f34dfdbbd354af68a1

TINI_VERSION 0.13.2

Plugins

Name ↓

Version

Enabled

ace-editor 1.1 true

ant 1.4 true

antisamy-markup-formatter 1.5 true

authentication-tokens 1.3 true

blueocean 1.0.0-b23 true

blueocean-autofavorite 0.6 true

blueocean-commons 1.0.0-b23 true

blueocean-config 1.0.0-b23 true

blueocean-dashboard 1.0.0-b23 true

blueocean-display-url 1.5.1 true

blueocean-events 1.0.0-b23 true

blueocean-git-pipeline 1.0.0-b23 true

blueocean-github-pipeline 1.0.0-b23 true

blueocean-i18n 1.0.0-b23 true

blueocean-jwt 1.0.0-b23 true

blueocean-personalization 1.0.0-b23 true

blueocean-pipeline-api-impl 1.0.0-b23 true

blueocean-rest 1.0.0-b23 true

blueocean-rest-impl 1.0.0-b23 true

blueocean-web 1.0.0-b23 true

bouncycastle-api 2.16.0 true

branch-api 2.0.6 true

build-pipeline-plugin 1.5.6 true

build-token-root 1.4 true

chromedriver 1.2 true

cloudbees-folder 5.17 true

compact-columns 1.10 true

conditional-buildstep 1.3.5 true

credentials 2.1.11 true

credentials-binding 1.10 true

cvs 2.13 true

dashboard-view 2.9.10 true

display-url-api 1.1.1 true

docker-commons 1.6 true

docker-workflow 1.10 true

durable-task 1.13 true

external-monitor-job 1.7 true

favorite 2.0.4 true

ghprb 1.35.0 true

git 3.0.5 true

git-client 2.2.1 true

git-notes 0.0.4 true

git-server 1.7 true

github 1.26.0 true

github-api 1.84 true

github-branch-source 2.0.3 true

github-oauth 0.25 true

github-organization-folder 1.6 true

gradle 1.26 true

groovy 1.30 true

handlebars 1.1.1 true

icon-shim 2.0.3 true

jackson2-api 2.7.3 true

javadoc 1.4 true

job-dsl 1.57 true

jquery 1.11.2-0 true

jquery-detached 1.2.1 true

junit 1.20 true

ldap 1.14 true

lockable-resources 1.11 true

mailer 1.19 true

mapdb-api 1.0.9.0 true

matrix-auth 1.4 true

matrix-project 1.8 true

maven-plugin 2.14 true

metrics 3.1.2.9 true

momentjs 1.1.1 true

naginator 1.17.2 true

nested-view 1.14 true

pam-auth 1.3 true

parameterized-trigger 2.32 true

pegdown-formatter 1.3 true

pipeline-build-step 2.4 true

pipeline-github-lib 1.0 true

pipeline-graph-analysis 1.3 true

pipeline-input-step 2.5 true

pipeline-milestone-step 1.3 true

pipeline-model-api 1.0.1 true

pipeline-model-declarative-agent 1.0.1 true

pipeline-model-definition 1.0.1 true

pipeline-rest-api 2.5 true

pipeline-stage-step 2.2 true

pipeline-stage-tags-metadata 1.0.1 true

pipeline-stage-view 2.5 true

plain-credentials 1.3 true

plot 1.9 true

pubsub-light 1.7 true

rake 1.8.0 true

ruby-runtime 0.13 true

rubyMetrics 1.6.5 true

run-condition 1.0 true

scm-api 2.0.4 true

script-security 1.26 true

simple-theme-plugin 0.3 true

slack 2.1 true

sse-gateway 1.15 true

ssh 2.4 true

ssh-agent 1.14 true

ssh-credentials 1.13 true

ssh-slaves 1.13 true

structs 1.6 true

subversion 2.7.1 true

throttle-concurrents 1.9.0 true

timestamper 1.8.8 true

token-macro 2.0 true

variant 1.1 true

windows-slaves 1.2 true

workflow-aggregator 2.5 true

workflow-api 2.11 true

workflow-basic-steps 2.4 true

workflow-cps 2.27 true

workflow-cps-global-lib 2.6 true

workflow-durable-task-step 2.9 true

workflow-job 2.10 true

workflow-multibranch 2.12 true

workflow-scm-step 2.3 true

workflow-step-api 2.9 true

workflow-support 2.13 true

xvnc 1.24 true
```