moby / moby/swarmkit

Race condition between task creation and removal

Open
#1,159 4 comments 1 reaction 1 assignee View on GitHub

@stevvooe is already working on this.

Since Jul 20, 2016.

area/agent kind/bug priority/P3
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

There exists a race condition between task creation and removal
which could causes orphan containers. It happens when a agent task is
handling creation event and is creating a container, a removal event
comes in and cancels the task's context, causing both the creation and
removal fail, but the container is still created by the docker daemon.
The removal fails because when the DELETE request arrives the docker
daemon, the container is still being created.

goroutine (creation)                   task (removal)
---------                              ----

...                                    ...
ctlr.Prepare(ctx)
  r.adapter.createNetworks(ctx)
  ...                                  case <-shutdown
  r.adapter.create(ctx)                  cancel()
  //failed due to context cancelled      tm.ctlr.Remove(ctx)
  //container still createdby daemon     //failed due to "No such
                                         //container" error because
                                         //the container not created
                                         //yet
  ...
  //the container created by docker
  //daemon but becomes orphan

There might be more race conditions between remove and other operations (e.g. Start) for the same task, but no harmful result has been observed so far.

This race issue is discussed in PR #1154, while the orphan container issue was originally reported by https://github.com/docker/docker/issues/24244.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.