packet loss after scrimlet swap

A customer reported packet loss from outside the rack to the API after having swapped a scrimlet with a different sled.

Observations by the time I got there:

• the actual internal DNS generation was 4
• the current target blueprint's internal DNS generation was 3
• blueprint execution was failing with a message about expecting the internal DNS generation to be 3, but found that it was 4
• DNS generation 4 contained Scrimlet DNS names (like Dendrite) only for one Scrimlet, not both

What we believe happened here is:

• the system was functioning fine with both scrimlets in-service with internal DNS generation 3 containing both scrimlets
• one scrimlet gets moved to a different slot and stops reporting itself as a scrimlet
• the next time a blueprint is executed (even if it's the same blueprint that's been there for a while), DNS generation 4 is created that removes the switch zone DNS names. This succeeds and propagates DNS version 4 out.
• another sled shows up in the former scrimlet slot
• again, the same blueprint is executed, but now Reconfigurator wants to create new DNS names for the sled that's now a scrimlet, but this time, it fails because the DNS generation has been bumped (by itself)

The system now comes to rest in the state that we found it: the switch zone services on one scrimlet are not known to the control plane, so we're not configuring Dendrite and friends, so any traffic flowing to that switch is probably not handled correctly.

There's a detailed comment about the generation numbers used here that clearly expects that the blueprint planner will create a new blueprint in this situation with the current internal DNS generation. However, the method that determines if a given blueprint is meaningfully different from the previous one ignores changes in the DNS generation. Our conclusion is that it shouldn't.

We believe this can be worked around by generating a new blueprint and making it the target. We were able to determine that generating a new blueprint created one whose only diff is the internal DNS generation number (which is good). It's to-be-confirmed whether making this the target is sufficient to repair the system.

More details coming.

Start by tracing the blueprint planner method that decides whether a blueprint is meaningfully different from the previous one, then read the detailed comment about internal DNS generation numbers. Confirm the generation-3-to-4 scenario and whether the DNS generation alone causes a new blueprint to be created. Done means the planner handles this change consistently and the reported packet-loss state can be validated.