[JENKINS-71349] git plugin left a lock file in controller cache folder for pipeline repos, which block all flows

git plugin left a lock file in controller cache folder for pipeline repos, which block all flows.

It happens after we uplift git plugin to 4.12.1.

I think it may related to the feature disable hooks or something else.

we do not enable hooks in controller.

It happens three times in one week.

I think sometime it could not release lock file success or it happens that multiple job try to lock same file.

When it happens, all job fails to load pipeline repos until we delete the lock file munally.

Could you please help to check it? Could we ignore this error when fail to change hooks configuration?

error log:
ERROR: Checkout failed
org.eclipse.jgit.errors.LockFailedException: Cannot lock /proj/XXX/workspace@​libs/72398dd74922d3156ed234ba0cbb097decd3eeabff521f0f59e1fa740478dc95/.git/config. Ensure that no other process has an open file handle on the lock file /proj/XXX/workspace@​libs/72398dd74922d3156ed234ba0cbb097decd3eeabff521f0f59e1fa740478dc95/.git/config.lock, then you may delete the lock file and retry.
at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:221)
at jenkins.plugins.git.GitHooksConfiguration.disable(GitHooksConfiguration.java:139)
at jenkins.plugins.git.GitHooksConfiguration.lambda$configure$2f1f6402$1(GitHooksConfiguration.java:113)
at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:87)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:112)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:99)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:91)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1240)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1312)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$doRetrieve$1(SCMSourceRetriever.java:201)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:148)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:200)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:137)
at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:260)
at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:150)
at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1087)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:513)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)

git plugin left a lock file in controller cache folder for pipeline repos, which block all flows.

It happens after we uplift git plugin to 4.12.1.

I think it may related to the feature disable hooks or something else.

we do not enable hooks in controller.

 

It happens three times in one week.

I think sometime it could not release lock file success or it happens that multiple job try to lock same file.

When it happens, all job fails to load pipeline repos until we delete the lock file munally.

Could you please help to check it? Could we ignore this error when fail to change hooks configuration?

 

error log:
ERROR: Checkout failed
org.eclipse.jgit.errors.LockFailedException: Cannot lock /proj/XXX/workspace@libs/72398dd74922d3156ed234ba0cbb097decd3eeabff521f0f59e1fa740478dc95/.git/config. Ensure that no other process has an open file handle on the lock file /proj/XXX/workspace@libs/72398dd74922d3156ed234ba0cbb097decd3eeabff521f0f59e1fa740478dc95/.git/config.lock, then you may delete the lock file and retry.
at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:221)
at jenkins.plugins.git.GitHooksConfiguration.disable(GitHooksConfiguration.java:139)
at jenkins.plugins.git.GitHooksConfiguration.lambda$configure$2f1f6402$1(GitHooksConfiguration.java:113)
at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:87)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:112)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:99)
at jenkins.plugins.git.GitHooksConfiguration.configure(GitHooksConfiguration.java:91)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1240)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1312)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$doRetrieve$1(SCMSourceRetriever.java:201)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:148)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:200)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:137)
at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:260)
at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:150)
at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1087)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:513)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)

• environment: Jenkins 2.346.1 with git plugin 4.12.1

Start with GitHooksConfiguration.configure and disable, especially the reported lines 99, 112, 113, and 139, then trace the repository wrapper through AbstractGitAPIImpl.withRepository and CliGitAPIImpl.withRepository. Reproduce the LockFailedException during GitSCM checkout for pipeline libraries and verify behavior around the .git/config.lock file; pull request #3972 is already open against this issue.