Service rm/scaledown in an old-worker w/ new manager cluster
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
https://github.com/docker/swarmkit/pull/2461 fixed an important bug, but introduced another one in a specific scenario that you can reproduce as follows:
- 2 node cluster: 1 manager and 1 worker with older swarmkit (aka before that PR)
- create a service with 5 replicas
- upgrade manager to newer (aka with that PR)
- remove service (or scaledown to 1)
The tasks are marked as REMOVED but no one is deleting the object, because the old worker doesn't understand the new REMOVED state and the manager no longer handles that.
This is caused because of the silent and then-innocuous bug in the agent code where the state was checked with == and not >=.
@aluzzardi suggests that for greater backwards compatibility, instead of changing an existing field we add a new Removed bool field that would behave similarly.
cc @stevvooe @nishanttotla @dperny @anshulpundir @marcusmartins @thaJeztah
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the two-node mixed-version cluster scenario described in the issue, starting with PR #2461 and the agent code's task-state check. Verify service removal or scale-down with an old worker, then ensure the REMOVED task objects are deleted while preserving compatibility with the older worker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100