dask / dask/distributed

GPU utilization not showing on MIG devices

Open
#8,739 0 comments 0 reactions 0 assignees View on GitHub
bug gpu
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

It was discovered while reviewing #8693 that GPUs partitioned with MIG do not support showing their utilization.

In #8693 I pushed a fix so that instead of getting an error when reading the utilization the graphs just show zero. But we may want to explore a longer term solution to getting MIG utilization.

The [NVML documentation states that utilization querying isn't supported on MIG](https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html#group__nvmlDeviceQueries_1g540824faa6cef45500e0d1dc2f50b321), and as far as I can see there is no alternative. If this is the case and it's not possible at all we may want to think about how we can communicate that to the user rather than just showing an empty graph.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.