Tracking Issue for Live Migration (Propolis)

Note: this issue is based on https://rfd.shared.oxide.computer/rfd/0071, and likely incomplete.
Please feel free to edit / modify it.

This issue attempts to, at a high-level, capture the necessary work items for LM. Depending on the size of sub-tasks, we can consider splitting out related issues, and dividing ownership.

• Protocol
  • Authenticated, encrypted TLS
  • Versions embedded in structures being transferred. We can start by being extremely selective, requiring exact version match.
  • Define Guest Machine Architecture (CPU features) and Platform (Device features) during instance creation
  • Advertise Guest Machine Architecture and Platform, ensure compatibility between source and destination. Related to https://github.com/oxidecomputer/omicron/issues/615
• State Management: Does Propolis' architecture expose necessary interfaces for quiescing, serializing, and transfer of devices?
• #82
  • Query / Offer dirty pages
  • Fetch full address space / Xfer full address space
  • Re-establish RAM in destination VM
• CPU Transfer
  • Registers
    • GP registers
    • MSRs
    • debug registers
    • control registers
  • Virtual APIC (ISA-defined registers, timers associated with registers, emulated acceleration (AVIC/APICv state)
  • Para-virtualized facilities (KVM or hyper-v para-virt clock)
  • TSC (offsets and freq) #359
  • HVM-specific state (interrupt injection, interrupt windowing, etcs)
    • "run-state", pending SIPI, etc
    • pending exception
    • pending interrupts
    • interrupt shadow #384
• In-Kernel Emulated Peripherals
  • PIT (i8254)
  • PIC (i8259)
  • IOAPIC
  • HPET
  • RTC
  • ACPI Power Management Timer
• Userspace Emulated Devices
  • Virtio Devices
  • NVMe
  • #98
• Miscellaneous Peripherals
  • Bootroom
  • Consumer-visible Interfaces
    • Framebuffer
    • KB / Mouse
    • ttya / ttyb
  • PCI state (hostbus, bridges)
• #84
  • "Deactivate" message sent to Crucible
  • Re-inflation of Crucible upstairs on target
• #83
  • "Pause" message sent to OPTE
  • Re-establishment of network traffic through OPTE

Related to https://github.com/oxidecomputer/omicron/issues/252 in Omicron

Start with RFD 0071 and this tracking issue's unchecked migration areas, especially protocol versioning and guest-machine compatibility. Review the linked Omicron issues and completed sub-issues to understand the existing scope. Done means the remaining live-migration checklist items are implemented or split into actionable issues, including CPU, peripheral, storage, and networking state.