Prevent from changing volume options (local driver) of a service when deploying a stack

Expected behavior

When deploying a stack to a Swarm, we should be informed that we are not allowed to change volume options (with local driver) of a service. This behavior is needed especially when you attach NFS volumes to your services, and you want to change one of these options: server address, remote path, mounting options... Which is a really common use case!

Maybe you could copy docker-compose's error message, which alreay handles this case:

ERROR: Configuration for volume foo specifies "o" driver_opt addr=192.168.77.254,ro,proto=tcp, but a volume with the same name uses a different "o" driver_opt (addr=192.168.77.254,rw). If you wish to use the new configuration, please remove the existing volume "foo" first:

Of course, it would be better to support volume options update. But since it would perform a umount/mount in case of NFS, I don't think it is possible. But maybe do you know some betters ways to share a NFS volume between service tasks? I know some plugins exist, but I wish Docker could handle this natively. Do you know if it is planned? What about being able to create "Swarm volumes", like overlay networks :D?

Today, to be able to update a volume options, you have to:

• use named volumes (https://docs.docker.com/compose/compose-file/#volume-configuration-reference : "name" field)
• when updating the volume options:
  • change its name
  • remove by hand the old volume on all nodes since docker volume prune does not work for NFS volumes: https://github.com/docker/for-linux/issues/389

This is very uncomfortable!

Actual behavior

When re-deploying a stack after changing volume options:

• we are not warned about it and the deploy command successes
• Case 1: the task remains on the same node: volume is not updated
• Case 2: the task is created on a another node, where the volume did not exist: the volume is created with the new options!
  • --> the behavior is thus unpredictable!

Steps to reproduce the behavior

Create this docker-compose.yml file:

---
version: "3.6"
services:
  test:
    image: alpine
    command: top
    volumes:
      - foo:/mnt/foo

volumes:
  foo:
    name: foo
    driver_opts:
      type: nfs
      o: addr=192.168.77.254,rw,proto=udp
      device: ":/var/exports/fornfsdev"

Deploy the stack:

➜  ~ docker stack deploy -c docker-nfs-issue/docker-compose.yml nfs-issue
Creating network nfs-issue_default
Creating service nfs-issue_test

Check that the volume has been created on the task node:

➜  ~ docker volume inspect foo
[
    {
        "CreatedAt": "2018-08-07T13:09:24+02:00",
        "Driver": "local",
        "Labels": {
            "com.docker.stack.namespace": "nfs-issue"
        },
        "Mountpoint": "/var/lib/docker/volumes/foo/_data",
        "Name": "foo",
        "Options": {
            "device": ":/var/exports/fornfsdev",
            "o": "addr=192.168.77.254,rw,proto=udp",
            "type": "nfs"
        },
        "Scope": "local"
    }
]

Then, replace o: addr=192.168.77.254,rw,proto=udp by o: addr=192.168.77.42,ro,proto=tcp in the docker-compose.yml and re-deploy the stack:

➜  ~ docker stack deploy -c docker-nfs-issue/docker-compose.yml nfs-issue
Updating service nfs-issue_test (id: 8pjcg8xggq6rfypc2xjsxw1h1)

The service has been updated, and the volume remains the same. And we have not been warned about this:

➜  ~ docker volume inspect foo                                           
[
    {
        "CreatedAt": "2018-08-07T13:09:24+02:00",
        "Driver": "local",
        "Labels": {
            "com.docker.stack.namespace": "nfs-issue"
        },
        "Mountpoint": "/var/lib/docker/volumes/foo/_data",
        "Name": "foo",
        "Options": {
            "device": ":/var/exports/fornfsdev",
            "o": "addr=192.168.77.254,rw,proto=udp",
            "type": "nfs"
        },
        "Scope": "local"
    }
]

Even worse, if the new task has been re-created on another node (where the volume did not exist), the volume has been created on this other node with the new options! So the behavior is totally unpredictable.

Output of docker version:

➜  ~ docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:09:54 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:07:56 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

➜  ~ docker info
Containers: 3
 Running: 2
 Paused: 0
 Stopped: 1
Images: 73
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: crers6copztuj09mxvcdosbvp
 Is Manager: true
 ClusterID: 8rn62v8lkdulsgrkn09ft3k64
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 127.0.0.1
 Manager Addresses:
  127.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-30-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.39GiB
Name: frioul
ID: NLDT:JDY2:RL36:DZZZ:LZIE:3WUW:6VEQ:OBNC:ZUA6:3BTV:AW7F:65UX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 http://127.0.0.1:5000/
Live Restore Enabled: false

WARNING: No swap limit support

Possible solution

I absolutely don't know how Docker works internally, but since docker service inspect returns the Mounts section (with volume options) in Spec and PreviousSpec, I suppose it would be possible to detect a volume options change, and stop the deploy.

Start with the docker stack deploy flow and compare the service inspect Mounts data in Spec and PreviousSpec, as suggested in the issue. Reproduce the NFS volume option change with the provided docker-compose.yml, then verify that redeployment detects the changed local-driver options and reports the condition instead of silently succeeding.