dask / dask/dask-cloudprovider
docker run --gpus=all will fail as nvidia-smi not available after bootstrap
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
During cluster bootstrap, the drivers are installed but they are not available as they are not loaded. It appears that a reboot must be done before `nvidia-smi` becomes available. As the nvidia drivers are not loaded, the command below will fail:
```
docker run --net=host --gpus=all ...
```
```python
from dask_cloudprovider.gcp import GCPCluster
def test_dask_gcp_cluster_gpu():
cluster = GCPCluster(
machine_type="n1-standard-8",
n_workers=1,
filesystem_size=100,
gpu_type="nvidia-tesla-t4",
ngpus=1,
)
```
`cloud-init-output.log`
```
Status: Downloaded newer image for daskdev/dask:latest
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
```
If GPUs are being used, the default image should already have drivers installed and useable or alternatively after driver install the nvidia driver should be loaded without requiring a reboot.
**Environment**:
- Dask version: 2022.9.2
- Python version: 3.10
- Operating System: ubuntu-os-cloud/global/images/ubuntu-minimal-1804-bionic-v20201014
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.