Validate NVML_FI_DEV_GET_GPU_RECOVERY_ACTION gpu state during health check
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 792
- Forks
- 109
- PR merge metrics
- No merged PRs in 30d
Description
It is appeared that DCGM is not able detect GPU which is stuck in EBUSY state because of pended recovery aciton
Example:
# GPU=1 stuck in EBUSY state because some GSP issue
nvidia-smi --query-gpu=index,gpu_recovery_action --format=csv,noheader
0, None
1, Reset <= GPU is unusable, reset required
2, None
3, None
4, None
5, None
6, None
7, None
dcgmi dmon -c 1 -e 230
#Entity XIDER
ID
GPU 0 0
GPU 1 120
GPU 2 0
GPU 3 0
GPU 4 0
GPU 5 0
GPU 6 0
GPU 7 0
This means that GPU is completely unusable, but health check report nothing
$ dcgmi health -s a
Health monitor systems set successfully.
$ dcgmi health -c -j
{
"body" :
{
"Overall Health" :
{
"value" : "Healthy"
}
},
"header" :
[
"Health Monitor Report"
]
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the health-check handling for NVML_FI_DEV_GET_GPU_RECOVERY_ACTION and compare it with the dcgmi health -c -j behavior shown in the report. Reproduce the EBUSY/Reset state with the supplied nvidia-smi and dcgmi commands; done means the health report identifies the GPU as unhealthy when recovery action is Reset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100