hashicorp / hashicorp/nomad

hot reconfiguration and live migration interface for task drivers

Open
#19,752 1 comment 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/driver type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal

Some types of task (both its in-memory state and its on-disk state) can be reconfigured or migrated to another node without it being shut down first. We should have a way to handle this gracefully.

We might need:

- A task interface to reconfigure the task without restarting it (if this interface returns a failure, fallback to the stop-then-start logic)
- A task interface to migrate the task's in-memory state to another node
- handle one task running on both nodes
- Some interface to migrate the volume to another node
- handle one volume on both nodes

### Use-cases

I'm investigating that if I can manage and monitor a bunch of VMs in Nomad with a custom task driver. These VMs might be stateless but I don't want them to be shutdown during a reschedule.

Other existing needs:

- Docker: https://github.com/hashicorp/nomad/issues/13785
- QEMU: https://github.com/hashicorp/nomad/issues/2323

and we might need: https://github.com/hashicorp/nomad/issues/15489

### Attempted Solutions

Currently Nomad only allows shutting down the task from the original node then starting it on the new node. The driver does not know on a high level that this task is being rescheduled rather than changed.

Writing a [remote task driver](https://github.com/hashicorp/nomad/issues/10549) on top of a current VM management solution (e.g. Proxmox VE) might be one possible way, but it is limited on my specific usage and does not scale well.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by examining the task-driver interface and the current stop-then-start rescheduling flow described in the issue. Compare the requested reconfiguration, task-state migration, and volume migration interfaces with the linked Docker and QEMU use cases. Done would require an agreed design for failure fallback and safe overlap on both nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.