[RFC] docker stack deploy should abort when failing to resolve the digest

While looking at some "stack deploy does not update to latest image" issues, I noticed that, if a registry is down, stack deploy will always fall back to using image:tag, likely leading to an outage of services.

Step 1 - prepare a registry

Start a local registry

docker run -d --name myregistry -p 5000:5000 registry:2

Push an image to the registry (re-tagging nginx:alpine for convenience)

docker image pull nginx:alpine
docker image tag nginx:alpine localhost:5000/myapp:latest
docker image push localhost:5000/myapp:latest

Remove the image locally to simulate a "fresh" node

docker image remove localhost:5000/myapp:latest nginx:alpine

Step 2 - deploy a stack

Deploy a stack that uses this image:

version: '3'
services:
  app:
    image: 'localhost:5000/myapp:latest'

docker stack deploy -c docker-compose.yml mystack

Verify that the service is using that image (and pinned by digest):

docker stack ps --no-trunc mystack

ID                          NAME                IMAGE                                                                                                 NODE                    DESIRED STATE       CURRENT STATE                ERROR               PORTS
yf80z6ce864nnd78g7mt7slmf   mystack_app.1       localhost:5000/myapp:latest@sha256:94a3611e66493d5903843b98427977ee083bdc6d7c4bda8d56d460a7c67a29ed   linuxkit-025000000001   Running             Running about a minute ago                       

Step 3 - simulate a registry outage

Stop the registry to simulate a situation where the registry could not be reached;

docker container stop myregistry

Re-deploy the stack

docker stack deploy -c docker-compose.yml mystack

Updating service mystack_app (id: uquxenc2vjtaoqq391v61wv96)
image localhost:5000/myapp:latest could not be accessed on a registry to record
its digest. Each node will access localhost:5000/myapp:latest independently,
possibly leading to different nodes running different
versions of the image.

The service is now down; even though connecting to the registry failed, docker updated the service definition, and changed localhost:5000/myapp:latest@<digest> to localhost:5000/myapp:latest.
This image is not present locally, causing the service to fail. If a local image was present, that image would've been used, potentially causing the service to be reverted to an old version of the image:

docker stack ps --no-trunc mystack

ID                          NAME                IMAGE                                                                                                 NODE                    DESIRED STATE       CURRENT STATE                     ERROR                                          PORTS
txdkbhi58lr9kfsd526c6tyxk   mystack_app.1       localhost:5000/myapp:latest                                                                           linuxkit-025000000001   Ready               Rejected less than a second ago   "No such image: localhost:5000/myapp:latest"   
w6419kh9h3zrsdsottldrj4fc    \_ mystack_app.1   localhost:5000/myapp:latest                                                                           linuxkit-025000000001   Shutdown            Rejected less than a second ago   "No such image: localhost:5000/myapp:latest"   
z1iupbhtxnvbgnpbfukendma1    \_ mystack_app.1   localhost:5000/myapp:latest                                                                           linuxkit-025000000001   Shutdown            Rejected less than a second ago   "No such image: localhost:5000/myapp:latest"   
sbj9j8ui7y2fnz2usp81cq2kr    \_ mystack_app.1   localhost:5000/myapp:latest                                                                           linuxkit-025000000001   Shutdown            Rejected 12 seconds ago           "No such image: localhost:5000/myapp:latest"   
yf80z6ce864nnd78g7mt7slmf    \_ mystack_app.1   localhost:5000/myapp:latest@sha256:94a3611e66493d5903843b98427977ee083bdc6d7c4bda8d56d460a7c67a29ed   linuxkit-025000000001   Shutdown            Shutdown 42 seconds ago                                    

Note that docker now paused roling out the service; other instances of the service will not be updated (limiting the outage)

"State": "paused",
"StartedAt": "2018-01-05T12:49:59.37566974Z",
"Message": "update paused due to failure or early termination of task pxep5sqyplppeum2ixr6vufua"

Step 4 - simulate the registry recovering

Start the registry again

docker container start myregistry

Docker is still trying to reconcile, and now succeeds to pull the image (by tag, not digest), and the service is back up again (but no longer pinned to a digest).

docker stack ps --no-trunc mystack

ID                          NAME                IMAGE                         NODE                    DESIRED STATE       CURRENT STATE                    ERROR                                          PORTS
6cgvjvcz9xdlxo3xktf7prwex   mystack_app.1       localhost:5000/myapp:latest   linuxkit-025000000001   Running             Running less than a second ago                                                  
zyv60tqwkjmluomwmfd2t49ox    \_ mystack_app.1   localhost:5000/myapp:latest   linuxkit-025000000001   Shutdown            Rejected 3 minutes ago           "No such image: localhost:5000/myapp:latest"   
zzm6wkprpiwi10c0wzihknlyo    \_ mystack_app.1   localhost:5000/myapp:latest   linuxkit-025000000001   Shutdown            Rejected 9 minutes ago           "No such image: localhost:5000/myapp:latest"   
zyo3ym1lxcmg2dpcuid6i3mr0    \_ mystack_app.1   localhost:5000/myapp:latest   linuxkit-025000000001   Shutdown            Rejected 20 minutes ago          "No such image: localhost:5000/myapp:latest"   
ztbpsmqoq9tvevycj8ozd7q8m    \_ mystack_app.1   localhost:5000/myapp:latest   linuxkit-025000000001   Shutdown            Rejected about an hour ago       "No such image: localhost:5000/myapp:latest"   

Re-deploy the stack to correctly pin the service to a digest again:

docker stack ps --no-trunc mystack

ID                          NAME                IMAGE                                                                                                 NODE                    DESIRED STATE       CURRENT STATE                     ERROR                                          PORTS
mhzjoqha4h17eiw5iif4kbrer   mystack_app.1       localhost:5000/myapp:latest@sha256:94a3611e66493d5903843b98427977ee083bdc6d7c4bda8d56d460a7c67a29ed   linuxkit-025000000001   Running             Running less than a second ago                                                   
6cgvjvcz9xdlxo3xktf7prwex    \_ mystack_app.1   localhost:5000/myapp:latest                                                                           linuxkit-025000000001   Shutdown            Shutdown less than a second ago                                                  
zyv60tqwkjmluomwmfd2t49ox    \_ mystack_app.1   localhost:5000/myapp:latest                                                                           linuxkit-025000000001   Shutdown            Rejected 4 minutes ago            "No such image: localhost:5000/myapp:latest"   
zzm6wkprpiwi10c0wzihknlyo    \_ mystack_app.1   localhost:5000/myapp:latest                                                                           linuxkit-025000000001   Shutdown            Rejected 10 minutes ago           "No such image: localhost:5000/myapp:latest"   
zyo3ym1lxcmg2dpcuid6i3mr0    \_ mystack_app.1   localhost:5000/myapp:latest                                                                           linuxkit-025000000001   Shutdown            Rejected 21 minutes ago           "No such image: localhost:5000/myapp:latest"   

Proposed solution

I think the current behavior is bad: Even though a warning is printed, by the time that warning is printed, it's too late, and the service has already been deployed. Falling back to image:tag should be a conscious decision.

Instead of automatically falling back to using image:tag, docker should abort deploying the service, and print an error instead, for example:

Updating service mystack_app (id: uquxenc2vjtaoqq391v61wv96)

image localhost:5000/myapp:latest could not be accessed on a registry to record
its digest. 

Use `--resolve-image=never` to skip resolving the digest, and deploy the service
using `localhost:5000/myapp:latest`.

Each node will access localhost:5000/myapp:latest independently, possibly
leading to different nodes running different versions of the image.

ping @dnephin @vdemeester PTAL

Start with the docker stack deploy image-resolution path and reproduce the registry outage using the commands and compose service shown in the issue. Trace how digest resolution failure changes the service definition. Done means deployment aborts with an error and explains --resolve-image=never, rather than falling back to the tag.