lablup / lablup/backend.ai

Add dcgm-exporter and node-exporter to halfstack as optional services

Open
#11,439 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Background

Following BA-5867 / BA-5870, the halfstack docker-compose has been split into required and optional services via Compose profiles (`telemetry`, `observability`, `storage`).

To make the local observability stack feature-parity with production monitoring, we should add two more exporters as optional services:

- **node-exporter** (`prom/node-exporter`) — host-level CPU / memory / disk / network metrics, used by the manager and ops dashboards in production
- **dcgm-exporter** (`nvcr.io/nvidia/k8s/dcgm-exporter`) — NVIDIA GPU metrics (utilization, memory, temperature, ECC errors), used to populate the GPU-related dashboards

## Scope

1. Add both services to `docker-compose.halfstack-main.yml` and `docker-compose.halfstack-ha.yml` under `profiles: ["observability"]`
1. Wire scrape targets in `prometheus.yaml` (gated so missing targets don't error when the profile isn't active)
1. Document in `.claude/skills/halfstack/SKILL.md` (services table + profile semantics)
1. Already covered by `--enable-observability` installer flag from BA-5870 since the flag activates the `observability` profile

## Notes

- Harbor halfstack integration is being tracked separately in PR #10963 (BA-5669) — do not duplicate here
- dcgm-exporter requires `nvidia-container-runtime` on the host; document this prerequisite in SKILL.md and skip gracefully if the host has no GPU
- Defaults remain off — these are opt-in via `--profile observability` (matches the BA-5867 / BA-5870 design)

## Acceptance criteria

- `docker compose -f docker-compose.halfstack.current.yml --profile observability up -d` brings up both new exporters along with the rest of the observability stack
- Prometheus successfully scrapes both targets
- Default (no profile) halfstack still starts without referencing either exporter

JIRA Issue: BA-5920

Contributor guide

Open the contributing guide

Research direction

Compare the observability services in docker-compose.halfstack-main.yml and docker-compose.halfstack-ha.yml, then read prometheus.yaml and .claude/skills/halfstack/SKILL.md. Verify the observability profile, Prometheus scrape behavior, and GPU prerequisite handling with the acceptance command, and confirm the default profile starts without either exporter.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, prometheus
Domain
devops, documentation, infrastructure, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.