hashicorp / hashicorp/nomad

Support canaries for task migrations

Open
#5,916 4 comments 17 reactions 0 assignees View on GitHub
theme/scheduling type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

It would be convenient if migrations (e.g., triggered by `nomad node drain` and governed by the `migrate` stanza) were able to create canary allocations. These allocations would be created before old allocations are destroyed, and old allocations would not be stopped until replacement allocations are deemed healthy. Unlike canaries, these replacements would probably not require manual promotion, but would instead promote automatically when healthy.

This feature would be useful to us since we run some workloads for staging or development purposes with `count = 1`. While these jobs do not require high-availability, it would nonetheless be preferable if services were not disrupted by a node drain operation. Since some of the applications we run require a 30-40 minute initialization time (I know, right) before service becomes available, the impact of a node drain in these cases can become a nuisance.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how `nomad node drain` handles the `migrate` stanza and how existing allocation canaries are created and promoted. Define the replacement lifecycle so new allocations start before old ones stop, become healthy automatically, and preserve service availability during long initialization; done means this behavior is covered for count=1 workloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.