moby / moby/swarmkit

Updater and restart supervisor can fight each other

Open
#2,242 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

priority/P3
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

  1. Suppose the update order is set to "start, then stop", and restart condition is set to "always". A change to the service starts a rolling update. During this rolling update, one of the existing tasks fails, and gets restarted. When the updater gets to this task, it won't be able to shut down the "old" task, because it was already restarted, but it will have already started a replacement task by this point. The service ends up with one too many replicas.
  2. Suppose the update order is set to "stop, then start", and restart condition is set to "none". A change to the service starts a rolling update. During this rolling update, one of the existing tasks fails. When the updater gets to this task, it fails to shut down the "old" task, so it does not start up a new one. The service ends up with fewer replicas than expected, even though the update should have restored the desired number of replicas.

This is tricky to fix because the updater is designed around being passed a list of tasks (which lets it operate on abstract slots, which are filled in by the orchestrator). I can see a few possible approaches:

  1. Disable task-level reconciliation during a rolling update, since the updater is effectively taking over this function. However, this would mean that a task that fails early on during a rolling update would not be restarted until the rolling update is entirely finished. Also, a rolling update technically lasts past the last task being replaced, because the updater stays around to monitor for failures.
  2. Change the restarter and orchestrator to reconcile the contents of a slot, instead of unconditionally updating or restarting. Note that updates already have this kind of logic, but it lives in the orchestrator rather than the updater so that the updater can operate on abstract slot objects.
  3. If any task in a service fails during an update, first restart it if applicable, then kick off a new rolling update to replace the current one, so it has an up-to-date list of tasks.

cc @aluzzardi

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the updater, restarter, and orchestrator reconciliation paths involved in rolling updates and task failures. Reproduce both update-order scenarios described in the issue and trace how task lists and abstract slots are handled. Done means failed tasks and rolling updates reconcile without extra or missing replicas.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.