moby / moby/swarmkit

Potential race condition in scheduler?

Open
#2,554 0 comments 0 reactions 2 assignees View on GitHub

@anshulpundir is already working on this.

Since Jun 5, 2018.

area/scheduler kind/bug
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

I noticed a few instances in manager/scheduler/scheduler.go where some checks seem suspect. In particular, we're doing something like

// Ignore all tasks that have not reached PENDING
		// state and tasks that no longer consume resources.
		if t.Status.State < api.TaskStatePending || t.Status.State > api.TaskStateRunning {
			continue
		}

in three places:

  1. https://github.com/docker/swarmkit/blob/master/manager/scheduler/scheduler.go#L71

  2. https://github.com/docker/swarmkit/blob/master/manager/scheduler/scheduler.go#L206

  3. https://github.com/docker/swarmkit/blob/master/manager/scheduler/scheduler.go#L243

From the comment, it seems like we're assuming that tasks with current state > RUNNING do not consume resources, which is no longer true with https://github.com/docker/swarmkit/pull/2461. For the brief period where the task reaper hasn't deleted some tasks, I'm trying to figure out if the task reaper could race with the scheduler here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.