Harden VMM lifecycle consistency across update, reload, and removal

Aperta
#767 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Refactoring
Chiarezza
Da chiarire
Stato di attività
Tranquilla
Stack tecnologico
rust

Direzione di ricerca

Inizia individuando gli entry point di UpdateVm, reload, removal, lifecycle serialization, manifest, disk-resize e CID-allocation descritti nell’issue. Traccia innanzitutto i relativi percorsi di mutazione e ripristino, quindi definisci il comportamento di coerenza e aggiungi test di injection dei guasti e delle operazioni concorrenti che coprano scritture parziali, riutilizzo di CID, race tra reload/removal e ripristino dopo un riavvio.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

Problem

VMM lifecycle operations update files, disk metadata, in-memory state, the CID pool, and supervisor processes across multiple steps. These steps are not consistently validated, serialized, or rolled back.

Review of #766 exposed the following independent issues. They are not part of the no-TEE feature and should be addressed separately.

Findings

  • UpdateVm writes the compose file, encrypted environment, and user config before later resource and manifest operations complete. A later failure leaves a partial update.
  • Disk resize occurs before manifest persistence and VM reload. A later failure can leave the qcow2 virtual size inconsistent with the manifest.
  • Manifest writes use a direct file write, so interruption can leave a truncated manifest.
  • storage_fs is derived from the image command line and app compose on each load. Changing either can change the expected filesystem for an existing disk.
  • Start, stop, update, reload, and removal are not serialized per VM. Concurrent operations can observe or overwrite intermediate state.
  • Reload rebuilds CID occupancy from running supervisor processes but does not preserve CIDs owned by stopped in-memory VMs. A new VM can reuse an existing CID. Supervisor processes excluded by annotation parsing may also fail to reserve their CIDs.
  • User removal awaits port-forward cleanup after writing .removing but before spawning background cleanup. Cancellation of the RPC future can leave removal marked but not progressing until reload or restart.
  • Orphan cleanup and reload can race: cleanup may finalize after reload has recreated state for the same VM and CID.

The existing .removing marker already provides restart recovery and should remain the durable source of removal intent.

Suggested direction

  • Validate the complete update before the first mutation.
  • Define transactional file and disk update behavior, including rollback or an ordering that cannot expose inconsistent state.
  • Serialize lifecycle operations per VM and define lock ordering for reload and CID allocation.
  • Rebuild CID ownership from both supervisor state and loaded stopped VMs, rejecting ownership conflicts.
  • Spawn removal cleanup before the RPC can be cancelled, while preserving .removing recovery.
  • Add failure-injection and concurrent-operation tests for partial writes, disk resize failure, CID reuse, reload/removal races, and process restart.

Context

These findings came from review of #766. The experimental fixes were removed from that PR to keep it scoped to development-only no-TEE support.

Lingua principale
Rust
Stelle
546
Fork
96
Merge medio
19h 22m
PR unite (30g)
109

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Dstack-TEE/dstack

Tutte le issue di Dstack-TEE/dstack

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.