[Question]: NIC resiliancy/failover and VNICs

Question

I have a few questions about NCCL failover feature and merged NICs, let me know what i got wrong in the first part where I explain my understanding.

Failover is expected to work when a send/recv communicator gets an error event and within it there is an extra device that can take the failed messages and the future traffic. Using single port NICs, by default, all these communicators have a single NIC, thus they cannot perform a failover. However we can merge two NICs into a merged VNIC and that might be used as a communicator for a channel.

If we have a DGXA100 with 8 GPUs and 8 NICs (single port) where 2 GPUs and 2 NICs share a PCIe switch we could merge the NICs from the same PCIe switch and theoretically failover would work. While testing this I've managed to get the 2 NICs to to merge but some channels still use the single VNICs while others use the merged ones. This causes failover to not be possible on this setup.

If we have a DGXH100 with the same configuration but each GPU has its own NIC then merging them would harm the common path as the merged VNIC will use mutliple PCIe switches. So failover with 2 NICs in this case is not optimal.

No source file or test is named. Start by reviewing NCCL's failover, communicator, channel, NIC, and merged-VNIC behavior described here, then determine whether mixed single-VNIC and merged-VNIC channel assignment is expected on DGXA100 and whether merging NICs across PCIe switches affects DGXH100 failover; done means documenting the confirmed behavior or a clearly scoped correction.