hashicorp / hashicorp/nomad

Unbound task creation lead to task stuck in Pending State

Open
#6,598 1 comment 1 reaction 0 assignees View on GitHub
theme/client theme/resiliency type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When starting tasks, drivers may not exercise care and make unbounded API calls that hang forever and lead to allocations getting stuck in pending stage. For example, docker driver calls docker.StartContainer[1] without passing a context or a timeout parameter; if docker agent hangs, we may . not recover and the allocation is stuck in pending state. The client may never recover and the allocation will never be restarted or rescheduled to another node.

Timeouts can be an effective technique here. By adding timeout, we can eventually recover and retry or reschedule. We can use a mixture of coarse grain timeouts (kill tasks if they take over 15 minutes to start regardless of specific case), and fine grain timeouts (allow for different timeouts for the fast StartContainer call and for the typically slow PullImage API calls).

Timeouts depend on many factors (e.g. system resources, ssd vs hdd, network speed, image size); so extra care is needed to balance fast detection of hang tasks vs allowing tail latencies of slow tasks. Should we go down this path, we should allow for sensible default timeout and possibilities to override them for individual tasks/clients/drivers.

Some prior art is the update stanza's [`healthy_deadline`](https://www.nomadproject.io/docs/job-specification/update.html#healthy_deadline) and [`progress_deadline`](https://www.nomadproject.io/docs/job-specification/update.html#progress_deadline).

[1] https://github.com/hashicorp/nomad/blob/v0.10.0/drivers/docker/driver.go#L454

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at drivers/docker/driver.go around the StartContainer call cited in the issue, and read the existing healthy_deadline and progress_deadline prior art. This issue first needs a decision on timeout scope, defaults, per-client or driver overrides, and recovery behavior before implementation can be considered complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.