jenkinsci / jenkinsci/docker-plugin
Pull from ECR never completes (no errors)
- Dominant language
- Java
- Stars
- 498
- Forks
- 324
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 2
Description
# Summary:
I am unable to pull from ECR - the pull is initiated but never completes and I can't see any errors.
Versions & Error Messages:
* docker-plugin version: 1.2.3 (from [pull/827](https://github.com/jenkinsci/docker-plugin/pull/827))
* jenkins version: 2.263.3
* docker engine version: 20.10.2
* java version: OpenJDK 1.11.0 (Java 11)
* OS: ubuntu-sever-20.04
* Images: jenkins/inbound-agent with internal CA added to truststore
```Dockerfile
FROM jenkins/inbound-agent
ENV USR_HOME=/home/jenkins
USER root
RUN mkdir -p "$USR_HOME/.cacerts"
COPY root_ca.crt "$USR_HOME/.cacerts/root_ca.crt"
RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit \
-import -noprompt -alias internal-ca -file "$USR_HOME/.cacerts/root_ca.crt"
USER ${user}
ENTRYPOINT ["/usr/local/bin/jenkins-agent"]
```
Stacktraces - there aren't any error messages, just the following:
```
2021-02-02 13:56:50.732+0000 [id=233] INFO c.n.j.plugins.docker.DockerCloud#canAddProvisionedAgent: Provisioning 'my-aws-account.dkr.ecr.eu-west-2.amazonaws.com/my-org/my-image:latest' on 'my-cloud'; Total containers: 0 (of 100)
2021-02-02 13:56:50.732+0000 [id=233] INFO c.n.j.plugins.docker.DockerCloud#provision: Will provision 'my-aws-account.dkr.ecr.eu-west-2.amazonaws.com/my-org/my-image:latest', for label: 'test-ecr', in cloud: 'my-cloud'
2021-02-02 13:56:50.734+0000 [id=233] INFO h.s.NodeProvisioner$StandardStrategyImpl#apply: Started provisioning Image of my-aws-account.dkr.ecr.eu-west-2.amazonaws.com/my-org/my-image:latest from my-cloud with 1 executors. Remaining excess workload: 0
2021-02-02 13:56:50.735+0000 [id=234] INFO c.n.j.p.docker.DockerTemplate#pullImage: Pulling image 'my-aws-account.dkr.ecr.eu-west-2.amazonaws.com/my-org/my-image:latest'. This may take awhile...
2021-02-02 13:56:50.742+0000 [id=234] INFO i.j.docker.client.DockerAPI#getOrMakeClient: Cached connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@8860c1b to DockerClientParameters{dockerUri=tcp://my-cloud:2376, credentialsId=my-creds, readTimeoutInMsOrNull=300000, connectTimeoutInMsOrNull=60000}
2021-02-02 13:59:21.149+0000 [id=236] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started DockerContainerWatchdog Asynchronous Periodic Work
2021-02-02 13:59:21.150+0000 [id=236] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog has been triggered
2021-02-02 13:59:21.151+0000 [id=236] INFO c.n.j.p.d.DockerContainerWatchdog$Statistics#writeStatisticsToLog: Watchdog Statistics: Number of overall executions: 3, Executions with processing timeout: 0, Containers removed gracefully: 0, Containers removed with force: 0, Containers removal failed: 0, Nodes removed successfully: 1, Nodes removal failed: 0, Container removal average duration (gracefully): 0 ms, Container removal average duration (force): 0 ms, Average overall runtime of watchdog: 55 ms, Average runtime of container retrieval: 53 ms
2021-02-02 13:59:21.151+0000 [id=236] INFO c.n.j.p.d.DockerContainerWatchdog#loadNodeMap: We currently have 0 nodes assigned to this Jenkins instance, which we will check
2021-02-02 13:59:21.151+0000 [id=236] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Checking Docker Cloud my-cloud at tcp://my-creds:2376
my-cloud-02-02 13:59:21.198+0000 [id=236] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog check has been completed
2021-02-02 13:59:21.199+0000 [id=236] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished DockerContainerWatchdog Asynchronous Periodic Work. 49 ms
2021-02-02 14:02:20.843+0000 [id=237] INFO hudson.model.AsyncAperiodicWork#lambda$doAperiodicRun$0: Started Update IdP Metadata from URL PeriodicWork
2021-02-02 14:02:20.846+0000 [id=237] INFO hudson.model.AsyncAperiodicWork#lambda$doAperiodicRun$0: Finished Update IdP Metadata from URL PeriodicWork. 1 ms
2021-02-02 14:04:21.150+0000 [id=238] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started DockerContainerWatchdog Asynchronous Periodic Work
2021-02-02 14:04:21.151+0000 [id=238] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog has been triggered
2021-02-02 14:04:21.152+0000 [id=238] INFO c.n.j.p.d.DockerContainerWatchdog$Statistics#writeStatisticsToLog: Watchdog Statistics: Number of overall executions: 4, Executions with processing timeout: 0, Containers removed gracefully: 0, Containers removed with force: 0, Containers removal failed: 0, Nodes removed successfully: 1, Nodes removal failed: 0, Container removal average duration (gracefully): 0 ms, Container removal average duration (force): 0 ms, Average overall runtime of watchdog: 53 ms, Average runtime of container retrieval: 51 ms
2021-02-02 14:04:21.152+0000 [id=238] INFO c.n.j.p.d.DockerContainerWatchdog#loadNodeMap: We currently have 0 nodes assigned to this Jenkins instance, which we will check
2021-02-02 14:04:21.152+0000 [id=238] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Checking Docker Cloud my-cloud at tcp://my-cloud:2376
2021-02-02 14:04:21.153+0000 [id=238] INFO i.j.docker.client.DockerAPI$1#entryDroppedFromCache: Dropped connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@8860c1b to DockerClientParameters{dockerUri=tcp://my-cloud:2376, credentialsId=my-creds readTimeoutInMsOrNull=300000, connectTimeoutInMsOrNull=60000}
2021-02-02 14:04:21.200+0000 [id=238] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog check has been completed
```
As you can see, the pull took over 20 minutes without completing. Pulling the image with the docker cli on my-cloud completed in under 20 seconds. Pulls of images on public DockerHub (no credentials) do succeed.
I'm using the Amazon ECR Credentials plugin (1.6).
## Request
It would be great if I could debug this in some way - it's frustrating to not know whats going wrong. I understand that I'm using an un-released/un-approved build but it does seem ulikely that anything in that PR caused this issue. So I guess What I'd like to understand:
1. Is this a known issue with a known fix?
2. Where can I get more information on what's going wrong to give me a fighting chance of fixing it?
As always, thanks for your work on this plugin - it's much appreciated.
Contributor guide
Assessment
This issue has not been assessed yet.