Slow docker commands with: credsStore and credHelpers

Description

Using credHelpers and credsStore with EG: ecr-login helper from AWS causes a big slowdown on docker build and other commands as it tries to log in to the registry every time, when sometimes credentials are not available for this.

Steps to reproduce the issue:

1. Configure ~/.docker/config.json EG:

{
	"credsStore": "secretservice",
	"credHelpers": {
		"123456.dkr.ecr.REGION.amazonaws.com": "ecr-login"
	}
}

2. Ensure you have no AWS credentials (we use aws-vault to load creds only when needed)
3. Setup a Dockerfile that DOES NOT use the registry. EG:

FROM openjdk:8-jre-slim
ENV JAVA_OPTS -XX:+CMSClassUnloadingEnabled -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm
WORKDIR /app

4. Run a docker build against said Dockerfile

Describe the results you received:

The build runs but only after ecr-login times out because there are no creds for it. I can confirm this is the case from the ecr-login helper logs:

time="2018-12-28T17:11:39+01:00" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2018-12-28T17:11:39+01:00" level=debug msg="Retrieving credentials" region=eu-central-1 registry=123123 serverURL=123123.dkr.ecr.eu-central-1.amazonaws.com
time="2018-12-28T17:11:39+01:00" level=debug msg="Calling ECR.GetAuthorizationToken" registry=123123
time="2018-12-28T17:12:00+01:00" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

Example run with secretservice example (or removing the credsStore line):

❯ time docker -D build -t test .
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM openjdk:8-jre-slim
 ---> 30bf70ee624c
Step 2/4 : ENV DEBIAN_FRONTEND noninteractiveback
 ---> Using cache
 ---> cc55395868a9
Step 3/4 : ENV JAVA_OPTS -XX:+CMSClassUnloadingEnabled -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm
 ---> Using cache
 ---> a9c8de304b1f
Step 4/4 : WORKDIR /app
 ---> Using cache
 ---> a45bdd081e42
Successfully built a45bdd081e42
Successfully tagged test:latest
docker -D build -t test .  0.05s user 0.04s system 0% cpu 40.856 total

Example run with pass example (or removing the credHelpers section):

❯ time docker -D build -t test .
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM openjdk:8-jre-slim
 ---> 30bf70ee624c
Step 2/4 : ENV DEBIAN_FRONTEND noninteractiveback
 ---> Using cache
 ---> cc55395868a9
Step 3/4 : ENV JAVA_OPTS -XX:+CMSClassUnloadingEnabled -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm
 ---> Using cache
 ---> a9c8de304b1f
Step 4/4 : WORKDIR /app
 ---> Using cache
 ---> a45bdd081e42
Successfully built a45bdd081e42
Successfully tagged test:latest
docker -D build -t test .  0.01s user 0.05s system 37% cpu 0.167 total

as you can see, both ran from cache, the wait time is on contacting the registries.

PS: I tried using the -D and -l "debug" to get more "verbose" output, but I cant find any additional log.

Describe the results you expected:

Since nothing is using the credHelpers specified registry and according to this code: https://github.com/docker/cli/blob/00e684311892ad63a8367066c6df4a5aa642o6507/cli/config/configfile/file.go#L289
it should not even try to execute the ecr-login helper and only use the default or the one specified on credsStore or that is my understanding from the docs linked below.

Additionally, if I set "credsStore": "pass", it works as expected, which is rather odd as pass is not even installed in my computer (helper nor CLI) and as expected docker login fails. Altho this might be a different bug.

I believe the culprit is https://github.com/docker/cli/blob/ea836abed5ba9c62c3d4444ea2a6bbf9b486ef1a/cli/command/image/build.go#L386 that calls GetAllCredentials instead of the creds required for the build only.

Additional information you deem important (e.g. issue happens only occasionally):

https://github.com/docker/cli/blob/254bcd27661e67707ac61f9532950e8ddec618a8/docs/reference/commandline/login.md

Output of docker version:

❯ docker version             
Client:
 Version:           18.09.0-ce
 API version:       1.39
 Go version:        go1.11.2
 Git commit:        4d60db472b
 Built:             Fri Nov  9 00:05:34 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.0-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.2
  Git commit:       4d60db472b
  Built:            Fri Nov  9 00:05:11 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

❯ docker info                                                                  
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 17
Server Version: 18.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m
runc version: 079817cc26ec5292ac375bb9f47f373d33574949
init version: fec3683
Security Options:
 seccomp
  Profile: default
 userns
Kernel Version: 4.19.12-zen1-1-zen
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.39GiB
Name: smallfish
ID: 6ISZ:WJFT:HQSZ:FXNB:EONI:RBIR:GKS4:5GWK:W4RY:WUJW:CCIT:5NPH
Docker Root Dir: /var/lib/docker/1000.1000
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

• Physical
• ECR Registry

Start with cli/command/image/build.go at the GetAllCredentials call and compare it with credential selection in cli/config/configfile/file.go. Reproduce the delay using the provided config.json, missing AWS credentials, and Dockerfile, then verify that a build only consults credentials for registries it needs and no longer waits on the unrelated ecr-login helper.