Unbound task creation lead to task stuck in Pending State
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
When starting tasks, drivers may not exercise care and make unbounded API calls that hang forever and lead to allocations getting stuck in pending stage. For example, docker driver calls docker.StartContainer[1] without passing a context or a timeout parameter; if docker agent hangs, we may . not recover and the allocation is stuck in pending state. The client may never recover and the allocation will never be restarted or rescheduled to another node.
Timeouts can be an effective technique here. By adding timeout, we can eventually recover and retry or reschedule. We can use a mixture of coarse grain timeouts (kill tasks if they take over 15 minutes to start regardless of specific case), and fine grain timeouts (allow for different timeouts for the fast StartContainer call and for the typically slow PullImage API calls).
Timeouts depend on many factors (e.g. system resources, ssd vs hdd, network speed, image size); so extra care is needed to balance fast detection of hang tasks vs allowing tail latencies of slow tasks. Should we go down this path, we should allow for sensible default timeout and possibilities to override them for individual tasks/clients/drivers.
Some prior art is the update stanza's [`healthy_deadline`](https://www.nomadproject.io/docs/job-specification/update.html#healthy_deadline) and [`progress_deadline`](https://www.nomadproject.io/docs/job-specification/update.html#progress_deadline).
[1] https://github.com/hashicorp/nomad/blob/v0.10.0/drivers/docker/driver.go#L454
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at drivers/docker/driver.go around the StartContainer call cited in the issue, and read the existing healthy_deadline and progress_deadline prior art. This issue first needs a decision on timeout scope, defaults, per-client or driver overrides, and recovery behavior before implementation can be considered complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100