Rebellions ATOM Max: total NPU power is overcounted 4x because per-card power is summed once per die
- Dominant language
- Rust
- Stars
- 279
- Forks
- 28
- Avg merge
- 8h 35m
- Merged PRs (30d)
- 30
Description
## Summary
On Rebellions ATOM Max, `all-smi` reports total NPU power at roughly 4x the real draw (measured 1369.7 W on a node whose NPUs actually draw 341.5 W). ATOM Plus is unaffected. The cause is that the Rebellions reader emits one device row per die while the power value it reads is a per-card figure, so every consumer that sums power counts each card four times.
## Background
`rbln-stat --json` enumerates dies, not cards. An 8-card ATOM Max node reports 32 devices (4 dies per card), and each die repeats its card's `card_power` value. ATOM Plus is one die per card, so summing per-die power happens to be correct there and the bug went unnoticed.
The reader confirms this. In `src/device/readers/rebellions.rs`, `get_npu_info_internal` iterates over `response.devices` (one entry per die) and calls `create_gpu_info_from_device` for each, which sets `power_consumption: parse_power_safe(&device.card_power)` (around line 396). One `GpuInfo` per die, each carrying the full per-card power.
That inflated value then flows into two surfaces:
1. TUI cluster summary: `total_power_watts` in `src/metrics/gpu_readings.rs` sums `power_consumption_reading` over every row unconditionally, and the result feeds `GpuClusterMetrics.total_power_watts` in `src/metrics/aggregator.rs`.
2. Prometheus: `all_smi_npu_power_watts` is emitted as one series per device, so `sum by (instance)` inflates by the same factor.
A side effect makes the average look right: `avg_power = total / reporting_count` = 1369.7 / 32 = 42.8 W, which coincidentally equals the true per-card power. So the average reads correctly while the total is 4x wrong.
Measured evidence from real captured output:
| | devices reported | physical cards | dies/card | total_power_watts | real NPU draw | overcount |
|---|---|---|---|---|---|---|
| ATOM Plus (RBLN-CA22) | 8 | 8 | 1 | 143.3 W | 143.3 W | 1.0x |
| ATOM Max (RBLN-CA25) | 32 | 8 | 4 | 1369.7 W | 341.5 W | 4.0x |
Per-die memory is not affected: it sums correctly today (32 x 15.72 GiB = 503 GiB, matching the hardware).
## Proposed Solution
A design decision is needed on how to model a multi-die card. Three options, from the report:
(a) Divide `card_power` by dies-per-card so each die carries a derived share. Total becomes correct, but it places a computed value in `power_consumption`, whose "absence" contract otherwise treats the field as directly measured.
(b) Report power on the first die of each card and leave it absent on the other three. `total_power_watts` already skips absent rows via `filter_map`, so the total becomes correct with all values measured. The cost is that 24 of 32 rows show no power, which can read as broken; the per-card value could still be carried in the `detail` map on every row for display.
(c) Collapse rows to cards (8 rows, memory summed per card). This is arguably the most faithful model, since the card is the unit the vendor's `rbln-stat` table prints power for and the unit Backend.AI allocates. The cost is that it changes the device count users see and requires summing per-die memory (which is correct as-is today).
Grouping key: `sid` (board serial) is the only portable grouping key that yields 8 groups on both Plus and Max. `group_id` collapses ATOM Plus to a single card, and `location` is a die-position-within-board code, not a card index, so neither works.
Maintainer input is requested on which option to adopt before implementation.
## Implementation Notes
- The per-die emission is in `create_gpu_info_from_device` and its caller `get_npu_info_internal` in `src/device/readers/rebellions.rs`. `card_power`, `sid`, and `location` are already parsed into `RblnDevice` there.
- Whichever option is chosen, the unconditional sum in `total_power_watts` (`src/metrics/gpu_readings.rs`) and its consumer `GpuClusterMetrics.total_power_watts` (`src/metrics/aggregator.rs`) should end up with a correct total without special-casing the vendor.
- Prometheus `all_smi_npu_power_watts` must not double-count once the fix lands.
- Option (b) relies on the existing absence encoding (`GPU_METRIC_UNAVAILABLE`) so that `filter_map` skips non-reporting rows.
- A captured `rbln-stat --json` from a real ATOM Max node is available to use as a test fixture.
## Acceptance Criteria
- On an 8-card ATOM Max node, the reported total NPU power matches the real draw (about 341.5 W in the captured sample), not 4x it.
- ATOM Plus behavior is unchanged (total stays 1.0x correct).
- Prometheus `all_smi_npu_power_watts` no longer double-counts per-card power when summed per instance.
- Per-die memory totals remain correct (no regression from any row collapsing).
- A regression test built from the ATOM Max fixture fails before the fix and passes after it.
---
## Original Suggestion
### Title: Rebellions ATOM Max: NPU power is reported at 4x the real draw (per-card value counted once per die)
## Summary
On Rebellions **ATOM Max**, `all-smi` reports NPU power at **4× the real draw** — 1369.7 W on a node whose NPUs actually draw 341.5 W.
`rbln-stat --json` enumerates **dies** (32 on an 8-card node), and `card_power` is a **per-card** value that all four dies of a card repeat. The reader emits one `GpuInfo` per die, so every consumer that sums `power_consumption` counts each card's power four times.
Measured from real captured output of an 8-card ATOM Max node (`RBLN-CA25`, KMD 3.0.0) and an 8-card ATOM Plus node (`RBLN-CA22`) for comparison:
| | devices reported | physical cards | dies/card | `total_power_watts` | real NPU draw | overcount |
|---|---|---|---|---|---|---|
| ATOM Plus | 8 | 8 | 1 | 143.3 W | 143.3 W | 1.0× |
| **ATOM Max** | **32** | **8** | **4** | **1369.7 W** | **341.5 W** | **4.0×** |
ATOM Plus is unaffected because it is one die per card, which is why this has gone unnoticed.
## Affected surfaces
1. **TUI cluster summary** — `src/metrics/gpu_readings.rs` sums unconditionally over every row:
```rust
pub fn total_power_watts(gpus: &[GpuInfo]) -> f64 {
gpus.iter().filter_map(GpuInfo::power_consumption_reading).sum()
}
```
consumed by `GpuClusterMetrics.total_power_watts` in `src/metrics/aggregator.rs`.
2. **Prometheus** — `all_smi_npu_power_watts` is emitted as one series per device, so `sum by (instance)` is inflated by the same factor.
Note a side effect: `avg_power = total / reporting_count` = 1369.7 / 32 = 42.8 W, which *coincidentally* equals the true per-card power. So the average reads correctly while the total is 4× wrong.
## How to identify a card
`sid` (board serial) is the only portable grouping key — it yields 8 groups on both Plus and Max. Two alternatives do **not** work:
- `group_id` (from `rbln-stat -g -j`) is `"1"` for all 8 devices on ATOM Plus, so grouping by it collapses Plus to a single card.
- `location` is a die-position-within-board code, not a card index — it is `5` for every device on Plus and cycles `1,2,3,4` within each card on Max.
Corroborating evidence that 8 is the card count: 8 distinct `sid` values (4 devices each), `rsd1..rsd8` in `/sys/class/rebellions/` each listing exactly its 4 `rbln` members, and 8 PCIe switches in `lspci -tv` each fronting 4 × `1eff:1250` endpoints.
## Possible fixes — I don't think this is mine to pick
| | total | per-row value | cost |
|---|---|---|---|
| **(a)** divide `card_power` by dies-per-card | correct | derived (~10.7 W) | puts a computed value in a field the absence contract treats as measured |
| **(b)** report power on the first die of each card, absent on the rest | correct (`filter_map` skips `None`) | all measured | 24 of 32 rows show no power, which reads as broken; the `detail` map could carry the card value on every row for display |
| **(c)** collapse rows to cards (8 rows, memory summed per card) | correct | all measured | loses per-die utilization/memory granularity |
**(c)** is arguably the most faithful model: the card is the coherent domain (the `topology` sysfs attribute is a 4×4 intra-card matrix), it is the unit Backend.AI allocates (`atom-max.device: 8`), and the vendor's own human-readable `rbln-stat` table prints Power once per 4-row block. But it changes the device count users see, and memory would need summing — per-die memory currently sums *correctly* (32 × 15.72 GiB = 503 GiB, matching the hardware), so that part is not broken today.
Happy to implement whichever you prefer, with an ATOM Max fixture and a test that fails before the fix. I have the captured `rbln-stat --json` from a real Max node to use as the fixture.
## Relationship to #416
Deliberately **not** folded into #416. That PR is a separate set of five silent-failure fixes, already green, and its tests are built from ATOM Plus output where this bug does not reproduce. This one needs a design decision plus a Max fixture.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with get_npu_info_internal and create_gpu_info_from_device in src/device/readers/rebellions.rs, then trace total_power_watts in src/metrics/gpu_readings.rs into src/metrics/aggregator.rs and the Prometheus output. Review the captured ATOM Max rbln-stat --json fixture and resolve which card-modeling option is accepted. Done means correct ATOM Max and unchanged ATOM Plus totals, preserved per-die memory totals, and a regression test covering the fixture.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100