Gradient Overflow with Specific GPU Combinations in Multi-GPU Setup (NVIDIA RTX 3090)

I'm facing an issue with gradient overflow when training a model using specific combinations of GPUs in a multi-GPU setup with 4 identical NVIDIA RTX 3090 GPUs on a single machine.

The issue occurs only when using GPU 2 and GPU 3 simultaneously. When I use GPUs 0, 1, and 2, or GPUs 0, 1, and 3, the training works fine. However, when using GPUs 2 and 3 together, I consistently encounter "Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to..." at the start of training.

!nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity
GPU0 X PXB PXB PXB 0-25,52-77 0
GPU1 PXB X PXB PXB 0-25,52-77 0
GPU2 PXB PXB X PIX 0-25,52-77 0
GPU3 PXB PXB PIX X 0-25,52-77 0

Changing the GPU combinations to avoid using GPUs 2 and 3 together solves the issue.
The problem seems tied to the PCIe connection between GPU 2 and GPU 3, but I’m not sure how to further address it.

No source file, test, or entry point is named. First reproduce the training failure with the reported GPU combinations and collect the relevant Apex, PyTorch, CUDA, and driver details; the issue is done when the cause is isolated to the software or PCIe setup and a verified fix or clear limitation is documented.