update system could use "disruption budget" to manage application disruption

Today, the self-service update process reboots instances many times as part of the update process as the underlying sleds get rebooted for host OS and SP updates. The live update project (RFD 665) seeks to mitigate this by live-migrating VMs to other sleds while updating the sled they were initially on. However, live migration is incompatible with local storage.

Consider the case of a distributed database deployed as a 9-node cluster that can tolerate (say) 3 simultaneous failures without compromising availability. Kubernetes supports the idea of a disruption budget that allows users to express this ("this app consists of 9 instances, of which you can disrupt as many as 3 at once"). Our update system could support something similar. One straw man is that you could attach this to an anti-affinity group. There'd likely need to be an associated health check. The update system would honor this, using the health check to verify the application before taking out any individual node and never taking out more than the specified number of nodes concurrently. (The health check is needed for the update system to know when a node that was disrupted is back in service, whatever that means for the application.)

This is basically what Reconfigurator already does with its own 5-node CockroachDB cluster.

This approach would allow us to manage update-related disruption for applications built to sustain some number of faults. It's potentially better than live migration (because LM is assumed to require some runtime impact that could at least briefly affect application latency) and it would work even for applications using local storage.

There are some product/feature definition questions here:

• What does the health check look like? For a disruption budget of 1, a binary "ok/fail" is enough. For a disruption budget of N > 1, we need something more fine-grained that can say "I'm not healthy because one node is gone, but I can sustain another failure, too". If it's an instance-level health check, maybe something like this? But it's not immediately obvious to me how to compose this.
• When exactly would we perform the health checks? How stale are they allowed to be? Do people need to be able to configure rules like "only mark failed if you fail 3 in a row; only mark healthy if you see 3 successful in a row"?
• Where is this configuration attached? Is the "group" always the anti-affinity group?
• Are we okay with an application being able to stall an update indefinitely by not responding to the health check (or reporting that it's not ready)? How would we mitigate this? Presumably we'd at least need to be able to surface this to operators and maybe give them a manual override.

Start by reading RFD 665 and the description of how Reconfigurator manages its CockroachDB cluster. Compare that behavior with Kubernetes disruption budgets, then define the health-check semantics, configuration attachment, update sequencing, staleness rules, and operator override behavior. Done means the product questions are resolved into an agreed feature design.