4paradigm / 4paradigm/k8s-vgpu-scheduler

commited image can not run in another node.

オープン
#8 コメント 3 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Go
スター
595
フォーク
100
PR マージ指標
30日以内にマージされた PR はありません

説明

_The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense._
---

### 1. Issue or feature description
commited image can not run in another node.
### 2. Steps to reproduce the issue
1. start pod with gpu enabled
2. commit container to image and push to registry
3. start pod with commited image in another node
container can not run with following error
```
Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook:
exit status 1, stdout: , stderr: nvidia-container-cli: device error: GPU-caba9b00-6386-2c33-7834-646ef2692cb7: unknown device\\\\n\\\"\"": unknown
```
### 3. Information to [attach](https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/) (optional if deemed irrelevant)

Common error checking:
- [ ] The output of `nvidia-smi -a` on your host
- [ ] Your docker configuration file (e.g: `/etc/docker/daemon.json`)
- [ ] The k8s-device-plugin container logs
- [ ] The kubelet logs on the node (e.g: `sudo journalctl -r -u kubelet`)

Additional information that might help better understand your environment and reproduce the bug:
- [ ] Docker version from `docker version`: 19.03
- [ ] Docker command, image and tag used: docker commit
- [ ] Kernel version from `uname -a`
- [ ] Any relevant kernel output lines from `dmesg`
- [ ] NVIDIA packages version from `dpkg -l '*nvidia*'` _or_ `rpm -qa '*nvidia*'`
- [ ] NVIDIA container library version from `nvidia-container-cli -V`
- [ ] NVIDIA container library logs (see [troubleshooting](https://github.com/NVIDIA/nvidia-docker/wiki/Troubleshooting))

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

The error indicates a GPU device mismatch when a committed container image runs on a different node. Start by examining the nvidia-container-cli logs and the device plugin's handling of GPU UUIDs. Look at the k8s-device-plugin container logs and the kubelet logs on the affected node to understand how GPU resources are allocated. Check if the committed image retains node-specific GPU identifiers that cause the 'unknown device' error.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
docker, go, kubernetes
領域
cloud, devops, infrastructure
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。