SDN20-02: Node rejoins a ready compute domain after a reboot (DRA)

Test statement

IMEX (Kubernetes DRA): After an unassisted reboot of a cluster node, the node rejoins a ready compute domain for the same workload automatically, with no tenant or operator intervention.

Source: docs/test-plan.yaml, SDN20-02.

What to assert

• Reboot affirmatively confirmed, by comparing boot identity or uptime across the reboot. Never infer that a reboot happened from reachability alone: a node that never went down is reachable too.
• After boot, and with no tenant or operator intervention: the node re-registers with the cluster and becomes schedulable, its NVLink clique label is republished, the domain's per-node daemon is rescheduled onto it, and a ready compute domain exists for the same workload.
• Report those four stages separately, each with elapsed time. A cluster that recovers everything except the clique label must fail for that specific reason rather than as a generic timeout, and a platform that recovers correctly but far too slowly must be visible.
• Assert that the node returns to a ready domain for the same workload, not that the original domain object survived. The driver may rebuild a domain rather than repair it, and a rebuilt ready domain satisfies the property.
• Any manual step needed to get there is a FAIL even if the end state is correct. Relabelling a node, restarting the driver, or recreating a claim by hand all count as intervention.
• Bounded but generous timeout, since GB200-class reboots take minutes and the domain re-forms only once the node is back.

Step output contract

The reboot comes from a provider step, since the cluster API cannot restart a node.
Every observation after it is a cluster API read. The shape below covers both.

{
  "success": true,
  "platform": "kubernetes",
  "node_id": "node-01",
  "reboot_confirmed": true,
  "boot_id_changed": true,
  "post_reboot": {
    "node_registered": true,
    "clique_label_republished": true,
    "daemon_rescheduled": true,
    "domain_ready": true,
    "elapsed_seconds": 214,
    "intervention_required": false
  }
}

The four post_reboot stage flags are each reported with their own elapsed time in
the check's subtest output, so a partial recovery names the stage that did not
complete.

Scope

In scope: rebooting one node of an allocated compute domain, and confirming that the node, its clique label, the domain's per-node daemon, and a ready domain for the same workload all return with no intervention.

Population: clusters where the resource-allocation driver owns the IMEX daemon. A cluster can instead be configured so that the driver defers to an operator-run host daemon and creates no per-domain daemon, which leaves the daemon-rescheduling stage above with no subject. Establish which ownership mode is in effect from the driver's declared configuration and record it, rather than inferring it from the absence of a daemon.

Out of scope: whole-cluster or whole-domain re-formation from cold. Rebooting every node is not an acceptable substitute for rebooting a single member.

Out of scope: survival of the original domain object. A rebuilt domain that is ready for the same workload satisfies this test.

Out of scope: the state, lifecycle, and configuration of any host IMEX daemon, including on a cluster where such a daemon legitimately exists and is required.

This test is destructive and slow. It reboots a node and must run only on an idle or drained domain. If it carries a slow label, that label must not exclude it from a certification run by default.

Notes

The reboot needs an out-of-band mechanism, since the cluster API cannot restart a node. Confirm which component republishes the NVLink clique label on the target platform, the device plugin or the driver, because that determines what a missing label indicts.