[reconfigurator] Should the rendezvous table RPW respect the "current target blueprint disabled" bit?

The current target blueprint has a boolean indicating whether or not it's enabled. The blueprint_executor task respects this bit by returning before doing any work if enabled is false. The blueprint_rendezvous task currently does not inspect this bit at all. It's not immediately obvious what the rendezvous task should do here. The below is an attempt to summarize a discussion we had about this.

The rendezvous task currently has two duties:

• If a debug dataset is in-service in the current target blueprint and is present in the latest inventory collection, ensure that a row for it exists in the debug_dataset rendezvous table.
• If a debug dataset is expunged in the current target blueprint, ensure that if a row for that dataset exists in the debug_dataset table, that row is tombstoned.

Failing to respect enabled=false presumably doesn't affect the first duty. Since it's guarded by "is present in inventory", a dataset can only be added if the blueprint was enabled at some point in the past (otherwise, how could the dataset exist to be present in inventory), so the rendezvous RPW could have seen it and written it at some point when the blueprint (or one of its ancestors) was enabled.

Failing to respect enabled=false does affect the second duty. If a new target blueprint is set with enabled=false which expunges a debug dataset, the rendezvous RPW will tombstone the row, preventing downstream consumers (e.g., support bundles) from using that dataset, even though the blueprint_executor will do nothing, presumably leaving the debug dataset in service until the blueprint is enabled. This is inconsistent and confusing; it means the current target blueprint can be partially acted upon regardless of its enabled status. In the specific case of debug datasets, the effect seems pretty minimal (we'll stop choosing a dataset that presumably will be expunged at some point in the future), but as the rendezvous RPW grows more this may become more difficult to reason about.

Changing the rendezvous RPW to respect enabled=false has a different downside: for now, we typically leave deployed systems with a disabled current target blueprint, but briefly enable blueprint execution during upgrades to step the system forward. If we change the rendezvous RPW to respect enabled=false, then the upgrade procedures that currently instruct an operator to confirm that execution is complete will also need to instruct them to confirm that updates to the rendezvous table are complete. (#7392 would help with this, but would be a nice improvement regardless of this issue.)

An aside that is maybe more important than the issue at hand: @smklein pointed out that setting a current target blueprint with enabled=false is confusing even without the above discussion. The current target blueprint is supposed to reflect the intended system state; what does it mean if that blueprint was never enabled (and thus blueprint_executor never even attempted to make the actual system state match the intended system state)?

In discussion we talked about (at least, I may be forgetting some) four options:

1. Keep the existing behavior. This seems fine in practice for now, but is confusing and might become less fine (or harder to reason about) as blueprint_rendezvous grows over time.
2. Change the RPW to respect enabled=false. This is easier to reason about but introduces the issue described above during upgrades (i.e., waiting for both execution and rendezvous work to complete before disabling the blueprint).
3. Fold the work of blueprint_rendezvous into blueprint_executor, so there's only one task. This seems dicey to me because it means we now need multiple passes of blueprint_executor to be done: a first pass that sends updating configs to sleds, and then a second pass later once any changes are reflected in inventory that updates rendezvous tables. I'm pretty strongly on team "these should be two separate tasks" (but this is certainly still up for discussion!).
4. Change the enabled flag: instead of being attached to the current target blueprint, attach it to specific RPWs. Then we can always say "the current target blueprint is the intended state of the system", but for debugging / support / recovery, we can disable blueprint_executor and/or blueprint_rendezvous via targeted flags. This also addresses the serious "aside" noted above, since we remove the concept of a disabled current target blueprint entirely.

My preference at the moment is option 4. If that's the way we go, I think the urgency of this is kinda low, because option 1 is equivalent to option 4 if we reinterpret the "current target blueprint enabled" field as "the blueprint_executor enabled field", but we should clean that up because it's misleading and confusing.

We also had a lot of discussion around terminology and what it means for a blueprint to be "executed" or "realized". A proposal for where we might land:

• "execute" and "execution" refer specifically to blueprint_executor; "execution is done" would mean "the blueprint_execution RPW completed"
• "realize" and "realized" refer to all system state related to blueprints - both blueprint_executor and blueprint_rendezvous; "the blueprint has been realized" would mean that blueprint_executor successfully executed it and that blueprint_rendezvous successfully updated any relevant rendezvous tables

Using these definitions:

• During on-site updates, we want to wait for blueprints to be realized, not just executed.
• If the enabled=false bit only applies to blueprint_executor, then a blueprint starts being realized as soon as it is the target (via the blueprint_rendezvous RPW potentially acting on it). However, as it is not executed, it will not finish being realized.

Start with the blueprint_executor and blueprint_rendezvous tasks described in the issue, and trace how the current target blueprint's enabled state affects each duty. Compare the four options, including upgrade implications and the proposed execution versus realization terminology. Done means a decided, consistent policy for realization and any identified follow-up changes.