jenkinsci / jenkinsci/ec2-plugin
[JENKINS-67714] Pipeline default Git checkout failing on EC2 Windows instances
- Dominant language
- Java
- Stars
- 292
- Forks
- 709
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 4
Description
We are seeing an intermittent issue where the first job running on a newly provisioned Windows EC2 agent fails to complete the automatic default checkout in the pipeline. The error shows there is an issue with the credentials being injected:
```
[2022-01-22T03:27:08.265Z] The recommended git tool is: git
[2022-01-22T03:27:10.101Z] using credential gerrit-jenkins-ssh
[2022-01-22T03:27:10.104Z] Wiping out workspace first.
[2022-01-22T03:27:12.284Z] Cloning the remote Git repository
[2022-01-22T03:27:12.878Z] Honoring refspec on initial clone
[2022-01-22T03:27:13.688Z] ERROR: Error cloning remote repo 'origin'
[2022-01-22T03:27:13.688Z] hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- ssh://mirror.eu-west-1.gerrit-eu01.aws.xxx.com:29418/xxxxxxxx/xxxxxxxx refs/heads/develop" returned status code 128:
[2022-01-22T03:27:13.688Z] stdout:
[2022-01-22T03:27:13.688Z] stderr: Warning: Identity file -o not accessible: No such file or directory.
[2022-01-22T03:27:13.688Z] ssh: Could not resolve hostname userknownhostsfile=/dev/null: Name or service not known
[2022-01-22T03:27:13.688Z] fatal: Could not read from remote repository.
[2022-01-22T03:27:13.688Z]
[2022-01-22T03:27:13.688Z] Please make sure you have the correct access rights
[2022-01-22T03:27:13.688Z] and the repository exists.
[2022-01-22T03:27:13.688Z]
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2661)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2086)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:86)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:608)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:837)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:160)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:153)
[2022-01-22T03:27:13.688Z] at hudson.remoting.UserRequest.perform(UserRequest.java:211)
[2022-01-22T03:27:13.688Z] at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2022-01-22T03:27:13.688Z] at hudson.remoting.Request$2.run(Request.java:376)
[2022-01-22T03:27:13.688Z] at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
[2022-01-22T03:27:13.688Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2022-01-22T03:27:13.688Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2022-01-22T03:27:13.688Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2022-01-22T03:27:13.688Z] at java.lang.Thread.run(Thread.java:748)
[2022-01-22T03:27:13.689Z] Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (Windows) - MobileStudio-WinServer2019-Agent-FVP (m5ad.4xlarge) (i-0d8427d26160b817e)
[2022-01-22T03:27:13.689Z] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
[2022-01-22T03:27:13.689Z] at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2022-01-22T03:27:13.689Z] at hudson.remoting.Channel.call(Channel.java:1001)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
[2022-01-22T03:27:13.689Z] at sun.reflect.GeneratedMethodAccessor1365.invoke(Unknown Source)
[2022-01-22T03:27:13.689Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2022-01-22T03:27:13.689Z] at java.lang.reflect.Method.invoke(Method.java:498)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
[2022-01-22T03:27:13.689Z] at com.sun.proxy.$Proxy112.execute(Unknown Source)
[2022-01-22T03:27:13.689Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1223)
[2022-01-22T03:27:13.689Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1301)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2022-01-22T03:27:13.689Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2022-01-22T03:27:13.689Z] ... 4 more
```
This looks to me like the variable containing the location of the SSH key file is empty.
As it is only intermittent, I thought it might be a timing issue but adjusting all available startup delays and timeouts has not helped.
Any idea or advice on how to diagnose what's going on here would be appreciated.
---
Originally reported by simepo, imported from: Pipeline default Git checkout failing on EC2 Windows instances
Raw content of original issue
We are seeing an intermittent issue where the first job running on a newly provisioned Windows EC2 agent fails to complete the automatic default checkout in the pipeline. The error shows there is an issue with the credentials being injected:
[2022-01-22T03:27:08.265Z] The recommended git tool is: git
[2022-01-22T03:27:10.101Z] using credential gerrit-jenkins-ssh
[2022-01-22T03:27:10.104Z] Wiping out workspace first.
[2022-01-22T03:27:12.284Z] Cloning the remote Git repository
[2022-01-22T03:27:12.878Z] Honoring refspec on initial clone
[2022-01-22T03:27:13.688Z] ERROR: Error cloning remote repo 'origin'
[2022-01-22T03:27:13.688Z] hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- ssh://mirror.eu-west-1.gerrit-eu01.aws.xxx.com:29418/xxxxxxxx/xxxxxxxx refs/heads/develop" returned status code 128:
[2022-01-22T03:27:13.688Z] stdout:
[2022-01-22T03:27:13.688Z] stderr: Warning: Identity file -o not accessible: No such file or directory.
[2022-01-22T03:27:13.688Z] ssh: Could not resolve hostname userknownhostsfile=/dev/null: Name or service not known
[2022-01-22T03:27:13.688Z] fatal: Could not read from remote repository.
[2022-01-22T03:27:13.688Z]
[2022-01-22T03:27:13.688Z] Please make sure you have the correct access rights
[2022-01-22T03:27:13.688Z] and the repository exists.
[2022-01-22T03:27:13.688Z]
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2661)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2086)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:86)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:608)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:837)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:160)
[2022-01-22T03:27:13.688Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:153)
[2022-01-22T03:27:13.688Z] at hudson.remoting.UserRequest.perform(UserRequest.java:211)
[2022-01-22T03:27:13.688Z] at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2022-01-22T03:27:13.688Z] at hudson.remoting.Request$2.run(Request.java:376)
[2022-01-22T03:27:13.688Z] at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
[2022-01-22T03:27:13.688Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2022-01-22T03:27:13.688Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2022-01-22T03:27:13.688Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2022-01-22T03:27:13.688Z] at java.lang.Thread.run(Thread.java:748)
[2022-01-22T03:27:13.689Z] Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (Windows) - MobileStudio-WinServer2019-Agent-FVP (m5ad.4xlarge) (i-0d8427d26160b817e)
[2022-01-22T03:27:13.689Z] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
[2022-01-22T03:27:13.689Z] at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2022-01-22T03:27:13.689Z] at hudson.remoting.Channel.call(Channel.java:1001)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
[2022-01-22T03:27:13.689Z] at sun.reflect.GeneratedMethodAccessor1365.invoke(Unknown Source)
[2022-01-22T03:27:13.689Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2022-01-22T03:27:13.689Z] at java.lang.reflect.Method.invoke(Method.java:498)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
[2022-01-22T03:27:13.689Z] at com.sun.proxy.$Proxy112.execute(Unknown Source)
[2022-01-22T03:27:13.689Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1223)
[2022-01-22T03:27:13.689Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1301)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
[2022-01-22T03:27:13.689Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2022-01-22T03:27:13.689Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2022-01-22T03:27:13.689Z] ... 4 moreThis looks to me like the variable containing the location of the SSH key file is empty.
As it is only intermittent, I thought it might be a timing issue but adjusting all available startup delays and timeouts has not helped.
Any idea or advice on how to diagnose what's going on here would be appreciated.
environment
```
Jenkins 2.289.1
ec2-plugin 1.62
Windows AMI Server 2019
```
Contributor guide
Assessment
This issue has not been assessed yet.