moby / moby/swarmkit

Service rm/scaledown in an old-worker w/ new manager cluster

Open
#2,494 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/orchestration kind/bug
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

https://github.com/docker/swarmkit/pull/2461 fixed an important bug, but introduced another one in a specific scenario that you can reproduce as follows:

  • 2 node cluster: 1 manager and 1 worker with older swarmkit (aka before that PR)
  • create a service with 5 replicas
  • upgrade manager to newer (aka with that PR)
  • remove service (or scaledown to 1)

The tasks are marked as REMOVED but no one is deleting the object, because the old worker doesn't understand the new REMOVED state and the manager no longer handles that.

This is caused because of the silent and then-innocuous bug in the agent code where the state was checked with == and not >=.

@aluzzardi suggests that for greater backwards compatibility, instead of changing an existing field we add a new Removed bool field that would behave similarly.

cc @stevvooe @nishanttotla @dperny @anshulpundir @marcusmartins @thaJeztah

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the two-node mixed-version cluster scenario described in the issue, starting with PR #2461 and the agent code's task-state check. Verify service removal or scale-down with an old worker, then ensure the REMOVED task objects are deleted while preserving compatibility with the older worker.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.