Add GPU Monitoring metrics
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Objective
Add GPU monitoring metrics to improve monitoring capabilities.
## Context
Currently, GPU monitoring metrics are available but not being utilized effectively.
This issue aims to enhance the handling of these metrics.
## Scope
- **Included**:
- Adding GPU monitoring metrics.
- Creating recovery scenarios based on the metrics.
- Implementing alerts if recovery is not feasible.
- Generating audit logs from GPU-related metrics.
- **Not Included**:
- Any metrics unrelated to GPU monitoring.
## Success criteria
- GPU monitoring metrics are successfully added.
- Recovery scenarios are documented.
- Alerts are implemented for difficult recovery situations.
- Audit logs are generated from GPU metrics.
## Risks and dependencies
- Potential challenges in implementing alerts.
- Dependency on the accuracy of the metrics provided by NVIDIA.
## Other information
- Reference for GPU monitoring metrics: [NVIDIA NVML API Documentation](https://docs.nvidia.com/deploy/nvml-api/group__nvmlEvents.html).
- The documentation describes methods that NVML can perform against each device to register and wait for events.
JIRA Issue: BA-1066
Contributor guide
Assessment
This issue has not been assessed yet.