lablup / lablup/backend.ai

Add GPU Monitoring metrics

Open
#4,074 0 comments 0 reactions 1 assignee Claimed by @kyujin-cho View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Add GPU monitoring metrics to improve monitoring capabilities.

## Context

Currently, GPU monitoring metrics are available but not being utilized effectively.
This issue aims to enhance the handling of these metrics.

## Scope

- **Included**:
- Adding GPU monitoring metrics.
- Creating recovery scenarios based on the metrics.
- Implementing alerts if recovery is not feasible.
- Generating audit logs from GPU-related metrics.
- **Not Included**:
- Any metrics unrelated to GPU monitoring.

## Success criteria

- GPU monitoring metrics are successfully added.
- Recovery scenarios are documented.
- Alerts are implemented for difficult recovery situations.
- Audit logs are generated from GPU metrics.

## Risks and dependencies

- Potential challenges in implementing alerts.
- Dependency on the accuracy of the metrics provided by NVIDIA.

## Other information

- Reference for GPU monitoring metrics: [NVIDIA NVML API Documentation](https://docs.nvidia.com/deploy/nvml-api/group__nvmlEvents.html).
- The documentation describes methods that NVML can perform against each device to register and wait for events.

JIRA Issue: BA-1066

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.