oxidecomputer / oxidecomputer/buildomat
want target for multi-node live migration jobs
@jordanhendricks is already working on this.
Since Apr 25, 2023.
- Dominant language
- Rust
- Stars
- 72
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
A concrete goal I have for propolis is running inter-machine live migrations of representative guest workloads in CI (oxidecomputer/propolis#360). The world I would like to move toward is collecting the various product assurance guest test configurations and workloads we have in mind (oxidecomputer/product-assurance#1 and https://github.com/oxidecomputer/product-assurance/issues/17 come to mind) and migrating guests running such workloads. Migrating guests with these workloads in CI will be very valuable on its own, but we could also include tests for performance regressions building off of existing work (oxidecomputer/propolis#347 and oxidecomputer/propolis#324 more generally).
While I am focused on propolis live migration specifically, in building this feature we will lay the groundwork for tests that will surely serve the rest of our upstack software, including omicron (even running omicron-drive migrations, perhaps) and crucible.
As a minimum starting point, I think a motivating example is doing a test I was doing a lot for the recent time-related live migration work: In a shell one-liner forever while loop, call a simple binary that reads the TSC, then sleeps for a second; migrate the guest and observe the guest TSC looks right on the other side. This test is nice because it doesn't require anything in the way of networking, only a disk, a serial console, and some minimal post-processing. For an image, we could use a file-backed disk that lives on the lab NFS share.
As for buildomat specifics, @jclulow and I chatted about some ideas here last week. At a minimum, we will need:
- a reserved 2-node cluster in the lab for this type of testing
- a new type of target in buildomat for multi-node jobs
- a basic set of synchronization primitives for scripts running on each machine. We will probably at least want a barrier primitive, so the two machines can wait at a given point before continuing.
The mechanics of how we orchestrate the migration mechanics I need to flesh out a bit more and will continue adding to this ticket as I do research and prototyping. One could imagine us using a script on one of the hosts to talk to the server API directly to coordinate setting up an instance (along with disks, vnics etc) and migrating it. Since we already have PHD, though, it would be nice to leverage that where we can. PHD today does not support networking I believe, so to do so we would need to add it. I am less certain of how easy it would be to modify PHD to do a server-like model here to coordinate some of the mechanics, but I am aware that @gjcolombo has thoughts.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.