NVIDIA / NVIDIA/DCGM

Validate NVML_FI_DEV_GET_GPU_RECOVERY_ACTION gpu state during health check

Open
#263 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
792
Forks
109
PR merge metrics
No merged PRs in 30d

Description

It is appeared that DCGM is not able detect GPU which is stuck in EBUSY state because of pended recovery aciton

Example:

# GPU=1 stuck in EBUSY state because some GSP issue
nvidia-smi --query-gpu=index,gpu_recovery_action --format=csv,noheader
0, None
1, Reset   <= GPU is unusable, reset required 
2, None
3, None
4, None
5, None
6, None
7, None

dcgmi dmon  -c 1 -e 230
#Entity    XIDER        
ID                      
GPU 0      0            
GPU 1      120          
GPU 2      0            
GPU 3      0            
GPU 4      0            
GPU 5      0            
GPU 6      0            
GPU 7      0   

This means that GPU is completely unusable, but health check report nothing

$ dcgmi  health -s a
Health monitor systems set successfully.
$ dcgmi  health -c -j
{
        "body" : 
        {
                "Overall Health" : 
                {
                        "value" : "Healthy"
                }
        },
        "header" : 
        [
                "Health Monitor Report"
        ]
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the health-check handling for NVML_FI_DEV_GET_GPU_RECOVERY_ACTION and compare it with the dcgmi health -c -j behavior shown in the report. Reproduce the EBUSY/Reset state with the supplied nvidia-smi and dcgmi commands; done means the health report identifies the GPU as unhealthy when recovery action is Reset.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.