hot reconfiguration and live migration interface for task drivers
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Proposal
Some types of task (both its in-memory state and its on-disk state) can be reconfigured or migrated to another node without it being shut down first. We should have a way to handle this gracefully.
We might need:
- A task interface to reconfigure the task without restarting it (if this interface returns a failure, fallback to the stop-then-start logic)
- A task interface to migrate the task's in-memory state to another node
- handle one task running on both nodes
- Some interface to migrate the volume to another node
- handle one volume on both nodes
### Use-cases
I'm investigating that if I can manage and monitor a bunch of VMs in Nomad with a custom task driver. These VMs might be stateless but I don't want them to be shutdown during a reschedule.
Other existing needs:
- Docker: https://github.com/hashicorp/nomad/issues/13785
- QEMU: https://github.com/hashicorp/nomad/issues/2323
and we might need: https://github.com/hashicorp/nomad/issues/15489
### Attempted Solutions
Currently Nomad only allows shutting down the task from the original node then starting it on the new node. The driver does not know on a high level that this task is being rescheduled rather than changed.
Writing a [remote task driver](https://github.com/hashicorp/nomad/issues/10549) on top of a current VM management solution (e.g. Proxmox VE) might be one possible way, but it is limited on my specific usage and does not scale well.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by examining the task-driver interface and the current stop-then-start rescheduling flow described in the issue. Compare the requested reconfiguration, task-state migration, and volume migration interfaces with the linked Docker and QEMU use cases. Done would require an agreed design for failure fallback and safe overlap on both nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100