[BUG] GCR authentication failure with docker_contain... | saltstack/salt

Nobody has claimed this yet.

Description
GCR authentication failure when using the docker_container.running state when run from the salt-master. Authentication appears to fail when running the docker pull command. I have attempted using this state module as both the root and docker-deploy user. If I run docker pull and docker run locally, with root or the deploy_user, it works as expected.

Setup

auth (init.sls)

add_auth_to_docker:
  cmd.run:
    - name: 'docker-credential-gcr configure-docker'
    - cwd: /home/<deploy_user>/
    - runas: <deploy_user>

init.sls

docker_container_name:
  docker_container.running:
    - image: gcr.io/<project>/<container>:latest
    - name:
    - user: <deploy_user>
    - ports_bindings:
    ▏ - <>:<>
    - restart_policy: always

Steps to Reproduce the behavior

sudo salt '<instance_name>*' state.apply docker_<env>

state.apply error

Failed to pull gcr.io/<project>/<container>:latest: Credentials store error: StoreError('Credentials store docker-credential-gcr exited with "".',)

OR

failed to find the <deploy_user> in the passwd file <-- salt appears to be looking for this
user on the salt-master

We have also seen the following error when changing the user to root

Failed to pull gcr.io/<project>/<container>:latest: Error 500: unauthorized: You don't have the needed permissions to perform this action

Expected behavior

I would expect the docker_container.running to pull the image from the private repo using the credentials supplied.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt-Master Version

Salt Version:
    ▏   ▏  Salt: 3000

Dependency Versions:
    ▏   ▏  cffi: Not Installed
    ▏  cherrypy: Not Installed
    ▏  dateutil: 2.6.1
    ▏ docker-py: Not Installed
    ▏   ▏ gitdb: 2.0.3
    ▏ gitpython: 2.1.8
    ▏   ▏Jinja2: 2.10
    ▏   libgit2: Not Installed
    ▏  M2Crypto: Not Installed
    ▏   ▏  Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
    ▏ pycparser: Not Installed
    ▏  pycrypto: 2.6.1
   pycryptodome: Not Installed
    ▏   ▏pygit2: Not Installed
    ▏   ▏Python: 2.7.17 (default, Nov  7 2019, 10:07:09)
   python-gnupg: 0.4.1
    ▏   ▏PyYAML: 3.12
    ▏   ▏ PyZMQ: 16.0.2
    ▏   ▏ smmap: 2.0.3
    ▏   timelib: Not Installed
    ▏   Tornado: 4.5.3
    ▏   ▏   ZMQ: 4.2.5

System Versions:
    ▏   ▏  dist: Ubuntu 18.04 bionic
    ▏   ▏locale: UTF-8
    ▏   machine: x86_64
    ▏   release: 5.0.0-1029-gcp
    ▏   ▏system: Linux
    ▏   version: Ubuntu 18.04 bionic

Salt-Minion Version

Salt Version:
    ▏   ▏  Salt: 3000.1

Dependency Versions:
    ▏   ▏  cffi: Not Installed
    ▏  cherrypy: Not Installed
    ▏  dateutil: 2.6.1
    ▏ docker-py: 4.2.0
    ▏   ▏ gitdb: Not Installed
    ▏ gitpython: Not Installed
    ▏   ▏Jinja2: 2.10
    ▏   libgit2: Not Installed
    ▏  M2Crypto: Not Installed
    ▏   ▏  Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
    ▏ pycparser: Not Installed
    ▏  pycrypto: 2.6.1
   pycryptodome: Not Installed
    ▏   ▏pygit2: Not Installed
    ▏   ▏Python: 2.7.17 (default, Nov  7 2019, 10:07:09)
   python-gnupg: 0.4.1
    ▏   ▏PyYAML: 3.12
    ▏   ▏ PyZMQ: 16.0.2
    ▏   ▏ smmap: Not Installed
    ▏   timelib: Not Installed
    ▏   Tornado: 4.5.3
    ▏   ▏   ZMQ: 4.2.5

System Versions:
    ▏   ▏  dist: Ubuntu 18.04 bionic
    ▏   ▏locale: UTF-8
    ▏   machine: x86_64
    ▏   release: 4.15.0-1026-gcp
    ▏   ▏system: Linux
    ▏   version: Ubuntu 18.04 bionic

Additional context

docker pull works for either user locally on the instance in question, the errors noted above only exist when trying to use the salt-master to provision the instance

Start by tracing the docker_container.running state entry point and the docker pull invoked during state.apply from the salt-master, comparing it with the cmd.run authentication setup. Done means a private GCR image can be pulled with the supplied credentials through docker_container.running, without the user lookup or authorization errors described.

Written by the indexing model from the issue text.

[BUG] GCR authentication failure with docker_container.running

Description

Contributor guide

First steps

Research direction

Assessment