Document rolling update process

We need to put up a doc on how rolling updates work with the operator. In the meantime, here is at least a dump of some discussion from Slack:

in general, all edits should be made only to the top-level vt. the others (vtc/vtk/vts) should be treated as read-only since they are actively managed by the top-level vt. this is similar to the pattern used in the core k8s Deployment API - one should not directly edit the underlying ReplicaSets it creates.

to create a new keyspace, you can just edit the vt spec to add it. it should get deployed immediately

however, edits to an already-deployed keyspace will be intentionally held back until the next rollout.

The rollout can be managed with annotations as follows:

whenever there are changes held back, the pending changes will be listed in an annotation on the object called rollout.planetscale.com/scheduled

so for example, when you edit an already-deployed keyspace in the vt, that annotation will appear on the corresponding vtk

the way to say "ok; go ahead and release those scheduled changes on this object" is to add an annotation called rollout.planetscale.com/released with any string value (even empty string). once this annotation is present, the scheduled changes will be applied

this needs to be done recursively: after releasing changes to the vtk, the vtk will schedule changes on its vts children which then need to be released; then each vts will schedule changes on the tablet Pods that will need to be released. the idea is that the operator handles actually doing the updates, but the process of deciding which things to update in which order and with what concurrency is left to an orthogonal system

btw, when the rollout.planetscale.com/released annotation is applied to a vttablet Pod with pending changes, the operator will first ensure that tablet is not a master for its shard, doing a PlannedReparentShard automatically if necessary, before recreating the Pod to update its spec

unless there are no other master-eligible replicas provisioned, in which case there's nothing that can be done except restarting the master. in that case, you must manually delete the master Pod once the rollout.planetscale.com/scheduled annotation appears on the Pod. the operator will not delete a master Pod.

Start with the rolling-update behavior and annotation names described in this issue, including the vt, vtk, vts, and vttablet Pod sequence. Document how scheduled changes are released, including the master-Pod exception, so a reader can follow the rollout order and know when the process is complete.