Service creation failure backoff-and-retry interval doesnt work for volume errors.
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
Steps to reproduce:
-
Initialize swarm on a node.
docker swarm init -
Create a node local volume.
docker volume create testvolume -
Now, create a service that uses
testvolume, but updates it with its own set of volume options.
docker service create --name testservice --replicas=1 --mount type=volume,volume-opt=o=addr=192.168.121.1,volume-opt=device=:/srv/nfs,volume-opt=type=nfs,source=testvolume,target=/nfs alpine sh -c 'while date; do sleep 5; done'
At the moment, this is allowed. However, this behavior is incorrect since volumes are immutable. So adding a patch such as the following, should error out the task creation for the service.
diff --git a/volume/store/store.go b/volume/store/store.go
index 4f4cffa..3d6a314 100644
--- a/volume/store/store.go
+++ b/volume/store/store.go
@@ -319,6 +319,7 @@ func (s *VolumeStore) Create(name, driverName string, opts, labels map[string]st
// TODO(cpuguy83): With v2 plugins this shouldn't be a problem. Could also potentially
// use a connect timeout for this kind of check to ensure we aren't blocking for a
// long time.
+
func (s *VolumeStore) checkConflict(name, driverName string) (volume.Volume, error) {
// check the local cache
v, _ := s.getNamed(name)
@@ -380,6 +381,8 @@ func volumeExists(store *drivers.Store, v volume.Volume) (bool, error) {
// If a volume with the name is already known, it will ask the stored driver for the volume.
// If the passed in driver name does not match the driver name which is stored
// for the given volume name, an error is returned after checking if the reference is stale.
+// If there are volume options and there's an existing volume by the given name, then
+// return error. Volumes are immutable, so dont update their options.
// If the reference is stale, it will be purged and this create can continue.
// It is expected that callers of this function hold any necessary locks.
func (s *VolumeStore) create(name, driverName string, opts, labels map[string]string) (volume.Volume, error) {
@@ -398,6 +401,10 @@ func (s *VolumeStore) create(name, driverName string, opts, labels map[string]st
return nil, err
}
+ if v != nil && len(opts) > 0 {
+ return nil, errors.New("Volume already exists. Cannot update volume with new options.")
+ }
+
if v != nil {
// there is an existing volume, if we already have this stored locally, return it.
// TODO: there could be some inconsistent details such as labels here
- With the patch in (3),
a. service creation will error out, as expected.
b. task creation will continue to be retried, as expected.
However, what is not expected is that there's no significant backoff. Task creation is retried several times per second. Trying the same operation at the same frequency, even if there are repeat failures doesn't seem like the right approach.
9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=0wvseumw41hqhspqlyobqpgka
time="2018-04-24T20:17:37.169003711Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=i4lwcq057314d6xl9jclaxanq
time="2018-04-24T20:17:37.422401740Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=y3cwz256itmn7no24wrd53nqn
time="2018-04-24T20:17:37.980495566Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=kspuaordfrtwxmhursywi1uwr
time="2018-04-24T20:17:38.241111984Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=z4g4ixgzh08jas7ncljbhdvgl
time="2018-04-24T20:17:43.257636689Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=ul5n20f8tqbaavfyvnt9uy406
time="2018-04-24T20:17:43.715472855Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=o4o43wolkycsfivkf0rcg2nsz
time="2018-04-24T20:17:43.968831289Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=ki7i0rfxsbcmb3nszwvegxfz1
time="2018-04-24T20:17:44.429070990Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=82qll18eas5xlk32oba16qf1k
time="2018-04-24T20:17:44.680785829Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=q9vhfh38bk7dsfhs2kn0rsrg1
time="2018-04-24T20:17:45.140651013Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=6i193tuaogjqr27gq8gurokyx
time="2018-04-24T20:17:45.397362558Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=mhscqs933qp8x31olcp0tvz1t
time="2018-04-24T20:17:45.857018316Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=m5062def7kn38l07hhjqxw5dj
time="2018-04-24T20:17:46.114124741Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=0axci90iy5l2q58qdgt4u39h8
time="2018-04-24T20:17:46.569823338Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=z9sutjj4ooypxc45oc5g6roo7
time="2018-04-24T20:17:46.823200870Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=5jvjzu69hyuozlarrant1ttyf
time="2018-04-24T20:17:47.278200943Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=1azlxta7v7vvwv235mh1bfz5d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with volume/store/store.go and trace the service task-creation path associated with the node/agent/taskmanager errors. Reproduce the immutable-volume failure from the documented Docker commands, then identify where retry timing is controlled; done means repeated failed task creation uses significant backoff rather than retrying several times per second.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100