Known issues with NCCL.

- 2.28 nccl doesn't have the `__file__` attribute for the nccl module. XGBoost from PyPI may fail to load nccl. (Workaround in XGBoost: https://github.com/dmlc/xgboost/pull/11806)
- Allgather inside XGBoost might fail when NCCL P2P is enabled, reproduced using 2.27. Workaround: disable NCCL P2P.
- 2.24.x and 2.25.x might crash XGBoost due to the RAS module. Fixed in 2.26
Workaround: `export NCCL_RAS_ENABLE=0`
* https://github.com/dmlc/xgboost/blob/3a2a85da38f8440903c2f5c06711be07c8143245/ops/pipeline/test-python-wheel-impl.sh#L48
* https://github.com/dmlc/xgboost/blob/461d27c1834c8dff42ce23b8c11b4de206cd71d2/ops/docker_run.py#L73

- Linux GPU driver 560 might hang p2p communication in nccl, or make it extremely slow.
Workaround:
* ~Update to 570~ (not solved) Disable P2P for now `NCCL_P2P_DISABLE=1`.
* Use `cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest` to check your p2p channel is working correctly.
* Check the correct BIOS configuration.

- With NCCL >= 2.24, you might run into an unhandled CUDA error reported by NCCL if your system doesn't support P2P communication.
Workaround:
Check `CUDA_VISIBLE_DEVICES` is correctly set. You should not split GPUs into different partitions:
* good: `CUDA_VISIBLE_DEVICES=1,0` to use the GPU with ordinal `1` as default (instead of `0`)
* bad: `CUDA_VISBLE_DEVICES=1`