bottlerocket-os / bottlerocket-os/bottlerocket
Tasks fail to start in g5g.xlarge instances for ECS NVIDIA variants
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
**Platform I'm building on:**
Any ECS NVIDIA variant, with the `g5g.xlarge` instance
**What I expected to happen:**
ECS tasks that requested NVDIA GPUs start in the node
**What actually happened:**
Tasks failed to start, and this message is printed:
```
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running
hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: driver rpc error: failed to process request: unknown.
```
This problem seems to be related to https://github.com/NVIDIA/nvidia-container-toolkit/issues/236. There are other solutions in the issue like starting enabling the persistence mode in the GPUs. I tried that, but the problem persisted. Another solution listed in the issue is to update the Docker version to something newer than v20. I haven't tried this since the ECS agent still requires docker v20.
**How to reproduce the problem:**
Launch any ECS NVIDIA variant on `g5g.xlarge` instances, and try to deploy a task with 1 GPU. The task will fail to start.
Contributor guide
Assessment
This issue has not been assessed yet.