aiondemand / aiondemand/aiod-enhanced-interaction
Fix GPU access on AIoD server
- 主要言語
- Python
- スター
- 0
- フォーク
- 1
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
On Merapi (GPU prod) server, Docker containers lose access to the GPUs after a certain amount of time passes...
For now we've circumvented this issue by including `nvidia-smi` as a part of the healthcheck.
Helpful links that may help when trying to fix this:
- https://github.com/NVIDIA/nvidia-container-toolkit/issues/381
- https://stackoverflow.com/questions/72932940/failed-to-initialize-nvml-unknown-error-in-docker-after-few-hours
One of possible culprits (also mentioned in the aforementioned Stackoverflow link):
- running this command `sudo systemctl daemon-reload` somehow disconnects containers from GPU
- I tested this myself and indeed after having run the command, containers lost access to GPU...
- Possible solution to test (not sure whether I should meddle with Docker settings though...): **Change cgroups**
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。