hashicorp / hashicorp/nomad

UI: Display GPU resource usage

Open
#10,779 11 comments 9 reactions 0 assignees View on GitHub
hcc/jira stage/needs-discussion theme/devices theme/ui type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal

According to #9917 it seems as though Nomad is already aware of certain resource usage information from GPUs, so I think that a simple solution would be to query for this data and display it using the existing graph components.

### Use-cases

As it stands, it is impossible to tell if a job is using GPU resources without SSH-ing into the machine and running a tool such as `nvidia-smi`. Even then, tools like `nvidia-smi` do not directly tell you which job is using the resources, only the process. It would be nice if the status page for the job itself could let you know of the GPU resource usage which, in turn, could help in visualizing when those resources get completely consumed (at least until a solution for #9917 becomes available).

### Attempted Solutions

None of the resource metrics available in the UI expose the GPUs at all. The closest I've seen to knowing that GPUs are available at all is the docker driver info on the client status page. Even then, that only shows that the nvidia runtime is available.

![2021-06-17-235123_867x442_scrot](https://user-images.githubusercontent.com/1240491/122503983-5bc4d580-cfc7-11eb-8dc3-cb2d6275a669.png)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the job status page and the existing graph components, then trace how current resource metrics are queried and how Docker driver information exposes GPU availability. Done means the UI shows GPU resource usage for the relevant job and makes consumption visible without requiring SSH or nvidia-smi.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
frontend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.