kvcache-ai / kvcache-ai/AgentENV

Add support for VM migration

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.5k
Forks
309
Avg merge
1d 9h
Merged PRs (30d)
66

Description

## Summary

Add the ability to move a sandbox from one AgentENV node to another. The core insight is that we already have fast snapshot and fast resume, so migration is mostly orchestration on top of existing primitives: capture (or reuse) a self-contained snapshot on the source node, make its artifacts reachable from the target node (P2P-first), resume on the target preserving sandbox identity, then tear down the source and hand off the scheduler binding so the gateway routes traffic to the new node.

Two cases, with different cost profiles:

1. **Running VM migration**: snapshot on source, resume on target. Reuses `capture_snapshot` plus create-from-snapshot.
2. **Paused VM migration**: the VM is already paused and its content sits node-local. We only need to publish that paused content into P2P so the target can pull and resume. No re-snapshot required.

## Motivation

Migration unlocks node draining for maintenance, load rebalancing, and bin packing. Paused sandboxes in particular are cheap to move because their state is already materialized, so making them mobile is high value for low cost.

## Proposed design

Introduce a migration flow (orchestrator + scheduler RPC) parameterized by source node, target node, and sandbox ID. Preserve the `SandboxId` end-to-end so the gateway keeps routing transparently after the binding flips.

**Scenario A: running VM**
1. On source: run `capture_snapshot` to produce a self-contained committed snapshot (recent `fix(sandbox): capture self-contained snapshot during pause` already moves us toward self-contained capture).
2. Ensure target reachability: rely on P2P advertisement of fixed artifacts + overlaybd layers.
3. On target: create-from-snapshot reusing the same `SandboxId`, wait for `Running`.
4. Flip scheduler binding to target (`RecordAssignment`), then delete the source sandbox.

**Scenario B: paused VM (P2P only, no re-snapshot)**
1. Add a publish-paused-to-P2P path: enumerate the paused VM's artifacts from its persisted `FirecrackerSnapshotConfig` (`vm_state.bin`, `mem_image.json` overlaybd layers including stacked ancestors, rootfs layers, writable drive layers) and `transport.publish` them. Reuse the content-addressed `overlaybd-layer/v1/sha256:` keys from `src/overlaybd/p2p/artifact.rs` for layers, and a `paused/v1/...` (or reuse `snapshot/v1/artifacts/...`) namespace for the fixed `vm_state.bin` + a reconstructable manifest.
2. On target: a resume-from-P2P resolver reconstructs `FirecrackerSnapshotConfig` from the published manifest and materializes every layer from P2P, then resumes.
3. Flip binding, then delete the paused source record and unpin/unpublish.

## Key challenges to resolve

- **P2P consumption gap**: P2P-first resolution today covers OSS fixed artifacts and the overlaybd layer facade. Migration without a shared repository backend (POSIX, or a paused VM that was never committed) needs the target to materialize *all* artifact classes from P2P. This is the main new capability for Scenario B.
- **Incremental memory + rootfs layers**: `mem_image.json` and rootfs are stacked overlaybd layers with parent ancestry. All ancestors must be reachable on the target. Content-addressed layer keys help, but the publish step must walk the full layer chain, not just the top layer.
- **Writable attached drives**: per-drive `drives//image.json` layers must migrate too.
- **Binding handoff atomicity**: traffic must flip to the target only after it is `Running`, and the source must not be deleted before the binding moves, to avoid a routing gap. Define the ordering and a rollback path if target resume fails.
- **CPU compatibility**: resume on a different host needs a compatible CPU config. The existing cluster-wide CPU config intersection (heartbeat `cpu_config_json`, applied via pre-boot `PUT /cpu-config`) is a precondition; migration should refuse or warn if the target cannot satisfy it.
- **Cleanup on both success and failure**: unpin/unpublish source artifacts, delete source sandbox/paused record, and handle partial failure (target resume fails, source already torn down).

## Suggested phasing

1. Paused VM publish-to-P2P + resume-from-P2P resolver (Scenario B), since paused content is already materialized and the cost/value ratio is best.
2. Running VM migration (Scenario A) layered on `capture_snapshot` + create-from-snapshot.
3. Scheduler/orchestrator migration RPC, binding handoff, and gateway transparency tests.

## Acceptance criteria

- A paused sandbox on node A can be resumed on node B with no shared repository backend, pulling all artifacts via P2P, preserving its `SandboxId`.
- A running sandbox can be migrated A -> B; the gateway routes client traffic to B after handoff with no manual reconfiguration.
- Migration failures leave exactly one live, routable copy (no orphaned source, no double bind).
- Integration coverage for both scenarios, including stacked memory/rootfs ancestry and writable drives.

Contributor guide

Open the contributing guide

Research direction

Start with src/overlaybd/p2p/artifact.rs, the existing capture_snapshot/create-from-snapshot flow, persisted FirecrackerSnapshotConfig, and the RecordAssignment scheduler handoff. Implement the paused publish-to-P2P and resume resolver before the running migration path. Done means both scenarios preserve SandboxId, pass integration coverage for stacked layers and writable drives, and leave one routable copy after failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.