`test_gpu_monitoring_recent` flaky
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
We observed `distributed/diagnostics/tests/test_nvml.py::test_gpu_monitoring_recent` fail in [this gpuCI build](https://gpuci.gpuopenanalytics.com/job/dask/job/distributed/job/prb/job/distributed-prb/332/CUDA_VER=11.2,LINUX_VER=ubuntu18.04,PYTHON_VER=3.8,RAPIDS_VER=21.10/console) over in https://github.com/dask/distributed/pull/5242
cc @charlesbluca who has experience with this part of the codebase
```python
12:46:45 =================================== FAILURES ===================================
12:46:45 __________________________ test_gpu_monitoring_recent __________________________
12:46:45
12:46:45 s =
12:46:45 a =
12:46:45 b =
12:46:45
12:46:45 @gen_cluster()
12:46:45 async def test_gpu_monitoring_recent(s, a, b):
12:46:45 if nvml.device_get_count() < 1:
12:46:45 pytest.skip("No GPUs available")
12:46:45
12:46:45 h = nvml._pynvml_handles()
12:46:45 res = await s.get_worker_monitor_info(recent=True)
12:46:45
12:46:45 > assert (
12:46:45 res[a.address]["range_query"]["gpu_utilization"]
12:46:45 == pynvml.nvmlDeviceGetUtilizationRates(h).gpu
12:46:45 )
12:46:45 E assert 2 == 0
12:46:45 E +2
12:46:45 E -0
12:46:45
12:46:45 distributed/diagnostics/tests/test_nvml.py:97: AssertionError
```
Contributor guide
Assessment
This issue has not been assessed yet.