google-deepmind / google-deepmind/dm_control

Support selecting an `EGL_DEVICE` by UUID rather than by index

オープン
#175 コメント 4 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
4.7k
フォーク
764
PR マージ指標
30日以内にマージされた PR はありません

説明

I am encountering an error with deploying `dm_control` in a managed HPC environment. Our admin decided to use `UUID` for the device names, which causes `dm_control` (and `mujoco-py`) to raise error when parsing the available devices:

```bash
Traceback (most recent call last):
File "/Users/ge/mit/dmc_gen/dmc_gen_analysis/__init__.py", line 164, in thunk
File "/home/gridsan/geyang/jaynes-mount/dmc_gen/2021-03-05/085031.707344/dmc_gen/dmc_gen/train.py", line 58, in train
image_size=image_size,
File "/home/gridsan/geyang/jaynes-mount/dmc_gen/2021-03-05/085031.707344/dmc_gen/dmc_gen/wrappers.py", line 28, in make_env
frame_skip=action_repeat
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dmc2gym/dmc2gym/__init__.py", line 55, in make
return gym.make(env_id)
File "/home/gridsan/geyang/.conda/envs/dmcgen/lib/python3.6/site-packages/gym/envs/registration.py", line 145, in make
return registry.make(id, **kwargs)
File "/home/gridsan/geyang/.conda/envs/dmcgen/lib/python3.6/site-packages/gym/envs/registration.py", line 90, in make
env = spec.make(**kwargs)
File "/home/gridsan/geyang/.conda/envs/dmcgen/lib/python3.6/site-packages/gym/envs/registration.py", line 59, in make
cls = load(self.entry_point)
File "/home/gridsan/geyang/.conda/envs/dmcgen/lib/python3.6/site-packages/gym/envs/registration.py", line 18, in load
mod = importlib.import_module(mod_name)
File "/home/gridsan/geyang/.conda/envs/dmcgen/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dmc2gym/dmc2gym/wrappers.py", line 2, in
from dm_control import suite
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dm_control/dm_control/suite/__init__.py", line 28, in
from dm_control.suite import acrobot
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dm_control/dm_control/suite/acrobot.py", line 24, in
from dm_control import mujoco
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dm_control/dm_control/mujoco/__init__.py", line 18, in
from dm_control.mujoco.engine import action_spec
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dm_control/dm_control/mujoco/engine.py", line 44, in
from dm_control import _render
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dm_control/dm_control/_render/__init__.py", line 67, in
Renderer = import_func() # pylint: disable=invalid-name
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dm_control/dm_control/_render/__init__.py", line 36, in _import_egl
from dm_control._render.pyopengl.egl_renderer import EGLContext
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dm_control/dm_control/_render/pyopengl/egl_renderer.py", line 69, in
EGL_DISPLAY = create_initialized_headless_egl_display()
File "/home/gridsan/geyang/mit/dmc_gen/custom_vendor/dm_control/dm_control/_render/pyopengl/egl_renderer.py", line 51, in create_initialized_headless_egl_display
devices = [devices[int(os.environ["CUDA_VISIBLE_DEVICES"])]]
ValueError: invalid literal for int() with base 10: 'GPU-a15dc796-f172-2e06-2283-cea8159bf118'
```

The reasoning behind this device `uuid` is explained in the following email (and issue)

> FYI, this is a known error where `dm_control` assumes CUDA_VISIBLE_DEVICES is an integer. We’re using NVIDIA’s UUID API to set the device names to the UUID, rather than the default. The problem with the default naming scheme (0,1,etc) is that it is not consistent. What’s listed as GPU 0 might change even within a job, which you can imagine would cause major problems if you have two people on a node, each allocated one GPU. This sort of alludes to what I’m talking about, but doesn’t get into using the UUID’s instead: https://stackoverflow.com/questions/26123252/inconsistency-of-ids-between-nvidia-smi-l-and-cudevicegetname. It’s a big oversight on Ray’s part to assume that the GPU names are integers, both Tensorflow and Pytorch don’t seem to have a problem with it. I think what they need to understand is that in a shared environment you have to make sure people use only the GPU that’s been allocated to them, and the way to do that is to use the UUID.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。