dask / dask/dask-cloudprovider

docker run --gpus=all will fail as nvidia-smi not available after bootstrap

Open
#393 2 comments 0 reactions 0 assignees View on GitHub
bug provider/gcp/vm
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

During cluster bootstrap, the drivers are installed but they are not available as they are not loaded. It appears that a reboot must be done before `nvidia-smi` becomes available. As the nvidia drivers are not loaded, the command below will fail:

```
docker run --net=host --gpus=all ...
```

```python
from dask_cloudprovider.gcp import GCPCluster

def test_dask_gcp_cluster_gpu():
cluster = GCPCluster(
machine_type="n1-standard-8",
n_workers=1,
filesystem_size=100,
gpu_type="nvidia-tesla-t4",
ngpus=1,
)

```

`cloud-init-output.log`

```
Status: Downloaded newer image for daskdev/dask:latest
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
```

If GPUs are being used, the default image should already have drivers installed and useable or alternatively after driver install the nvidia driver should be loaded without requiring a reboot.

**Environment**:

- Dask version: 2022.9.2
- Python version: 3.10
- Operating System: ubuntu-os-cloud/global/images/ubuntu-minimal-1804-bionic-v20201014
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.