bottlerocket-os / bottlerocket-os/bottlerocket

Tasks fail to start in g5g.xlarge instances for ECS NVIDIA variants

Open
#3,513 3 comments 2 reactions 1 assignee Claimed by @arnaldo2792 View on GitHub
area/accelerated-computing status/research type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**Platform I'm building on:**
Any ECS NVIDIA variant, with the `g5g.xlarge` instance

**What I expected to happen:**
ECS tasks that requested NVDIA GPUs start in the node

**What actually happened:**
Tasks failed to start, and this message is printed:

```
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running
hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: driver rpc error: failed to process request: unknown.
```

This problem seems to be related to https://github.com/NVIDIA/nvidia-container-toolkit/issues/236. There are other solutions in the issue like starting enabling the persistence mode in the GPUs. I tried that, but the problem persisted. Another solution listed in the issue is to update the Docker version to something newer than v20. I haven't tried this since the ECS agent still requires docker v20.

**How to reproduce the problem:**

Launch any ECS NVIDIA variant on `g5g.xlarge` instances, and try to deploy a task with 1 GPU. The task will fail to start.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.