dask / dask/distributed

`test_gpu_monitoring_recent` flaky

Open
#5,287 6 comments 0 reactions 0 assignees View on GitHub
flaky test
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

We observed `distributed/diagnostics/tests/test_nvml.py::test_gpu_monitoring_recent` fail in [this gpuCI build](https://gpuci.gpuopenanalytics.com/job/dask/job/distributed/job/prb/job/distributed-prb/332/CUDA_VER=11.2,LINUX_VER=ubuntu18.04,PYTHON_VER=3.8,RAPIDS_VER=21.10/console) over in https://github.com/dask/distributed/pull/5242

cc @charlesbluca who has experience with this part of the codebase

```python
12:46:45 =================================== FAILURES ===================================

12:46:45 __________________________ test_gpu_monitoring_recent __________________________

12:46:45

12:46:45 s =

12:46:45 a =

12:46:45 b =

12:46:45

12:46:45 @gen_cluster()

12:46:45 async def test_gpu_monitoring_recent(s, a, b):

12:46:45 if nvml.device_get_count() < 1:

12:46:45 pytest.skip("No GPUs available")

12:46:45

12:46:45 h = nvml._pynvml_handles()

12:46:45 res = await s.get_worker_monitor_info(recent=True)

12:46:45

12:46:45 > assert (

12:46:45 res[a.address]["range_query"]["gpu_utilization"]

12:46:45 == pynvml.nvmlDeviceGetUtilizationRates(h).gpu

12:46:45 )

12:46:45 E assert 2 == 0

12:46:45 E +2

12:46:45 E -0

12:46:45

12:46:45 distributed/diagnostics/tests/test_nvml.py:97: AssertionError
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.