AI-Hypercomputer / AI-Hypercomputer/xpk

NCCL is installed on cluster GPU hosts by default

未關閉
#1,014 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
193
分支
94
平均合併
24 分鐘
30 天內合併 PR
1

描述

`NCCL` is installed on cluster GPU hosts by default (e.g on `a3-megagpu-8g` clusters) while this may not be the most desirable default behaviour.

User-containers (the '`gpu-image`' in `xpk`) typically bundle their own `NCCL` version that user code depends on and expects to use. However, in GKE we have found that this expectation is broken because `NCCL` binaries from GPU hosts are mounted into the containers and take precedence in the `LD_LIBRARY_PATH` by default. This could cause user code to break if there are discrepancies between the container and host NCCL versions.

It would be helpful to be able to easily disable `NCCL` install on GPU hosts by the `nccl-tcpxo-installer` pods. The manifest provided to `xpk` from `container-engine-accelerators` provides a way to do this by removing the `-install-nccl` flag ([nccl-tcpxo-installer.yaml#L88](https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/master/gpudirect-tcpxo/nccl-tcpxo-installer.yaml#L88))

With `xpk` cluster creation I was only able to do this by manually editing the deployment blueprints after being generated by xpk (and then manually deploying that). I was not able to get an end-to-end cluster deployment with `xpk` by, for example, repointing the hardcoded paths to the manifest ([e.g.](https://github.com/AI-Hypercomputer/xpk/blob/release-1.0/src/xpk/core/system_characteristics.py#L28)).

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。