lablup / lablup/all-smi

test: verify IOReport energy channels and cadence on M1 to M4 and Ultra

Open
#415 0 comments 0 reactions 0 assignees View on GitHub
device:apple-silicon priority:medium status:ready type:test
Dominant language
Rust
Stars
279
Forks
28
Avg merge
8h 35m
Merged PRs (30d)
30

Description

## Problem / Background

#410 (PR #412, merged `c5bf759`) computes Apple Silicon power from each `Energy Model` channel's driver publication timestamp (byte offset 24 of the channel's `RawElements`, xnu `IOReportElement`) and matches channels exactly in `classify_energy_channel` (`src/device/macos_native/energy.rs:75-89`):

| rail | channels |
|---|---|
| CPU | names ending in `CPU Energy`, including `DIE__CPU Energy` |
| GPU | `GPU Energy`; `GPU` only when a sample has no `GPU Energy` |
| ANE | top-level `ANE`, `ANE`, `ANE_` |
| DRAM | top-level `DRAM`, `DRAM`, `DRAM_` |

It was verified only on an Apple M5 Max (Mac17,7, macOS 27.0): 364-channel fixture `tests/fixtures/ioreport/m5_max_energy_model.tsv`, mJ batches 2.03 to 2.23 s apart, split tails 9 to 26 ms, `GPU Energy` stamped at sample time and exactly 0 at idle. A channel whose timestamps are missing, frozen while the value moves, or more than 1 s ahead of the observation falls back to observation time permanently (`ChannelState::observe`, `energy.rs:238-297`).

Unknown on other chips: the channel names (Ultra die prefixes on GPU, ANE, and DRAM; a bare `GPU` channel, which classifies as nothing today), whether the counters are batched, and whether the drivers stamp the element timestamps. GitHub macOS runners are VMs without IOReport, so this needs real hardware; the maintainer has an M1 Ultra.

## Current Behavior

- `ioreport_energy_diagnostics` (`src/device/macos_native/ioreport.rs:1725-1827`, ignored, macOS only) prints the whole `Energy Model` inventory in the fixture format, enumerated from the group rather than the subscription, then 6 s of 100 ms samples with each tracked channel's counter delta, publication span, and publication age next to the rail readings.
- The cadence part covers only channels `classify_energy_channel` accepts, because the subscription is filtered by the same predicate (`keep_energy_channel`, `src/device/macos_native/ioreport/channel_filter.rs:57`). A rail channel the classifier misses is neither subscribed nor summed, so it reads as 0 W.
- A package `CPU Energy` next to `DIE__CPU Energy` would match twice and be summed; `multi_die_rails_are_summed` (`src/device/macos_native/energy/tests.rs:210`) assumes only per-die CPU channels exist. `DIE__`-prefixed GPU, ANE, or DRAM channels match no rail.
- SMC temperatures (`src/device/macos_native/smc.rs:798-876`) average the in-range static keys per rail (CPU `Tp01 Tp02 Tp05 Tp06 Tp09 Tp0A TC0P TC0D`, GPU `Tg0f Tg0j TG0P TG0D`) and fall back to discovered keys only when no static key reads, so aggregation differs by chip: on the M5 Max the GPU temperature comes from the single static key `Tg0j` while the CPU temperature averages 23 discovered keys, and on chips without static keys the GPU temperature averages every discovered GPU sensor. The diagnostic prints none of this.

## Proposed Solution

1. Run `cargo test --lib ioreport_energy_diagnostics -- --ignored --nocapture` on an M1 Ultra at minimum and on any available M1 to M4 machine, idle and under load.
2. Add each inventory as `tests/fixtures/ioreport/_energy_model.tsv` (for example `m1_ultra_energy_model.tsv`) with a header comment in the M5 Max file's format (chip, model identifier, macOS build, capture date), plus classification tests like `m5_max_inventory_classifies_only_the_rail_channels` and `m5_max_loaded_batch_matches_the_roll_up` (`energy/tests.rs:101`, `:156`) that pin which channels feed each rail.
3. Re-check the #410 load behavior on each chip: CPU power equals the `CPU Energy` roll-up or the sum of `DIE__CPU Energy`, no 0 W ticks at `--interval 1`, GPU counted once.
4. Decide and implement `DIE__` handling for GPU, ANE, and DRAM from the Ultra inventory, in `classify_energy_channel` and `sum_rails` (`energy.rs:144`). A per-die sum must not double count a package total: where both exist, exactly one of them feeds the rail. Fix any other rule a chip contradicts.
5. Extend the same diagnostic run to print the SMC temperature key inventory: which static CPU and GPU keys exist and read in range, how many CPU and GPU keys discovery finds (`SMC::scan_temperature_keys`, `smc.rs:489`), and which path each rail takes. Record the results in this issue.

## Scope

**In scope:** new fixtures under `tests/fixtures/ioreport/`, `src/device/macos_native/energy.rs` and `energy/tests.rs`, the diagnostic and module docs in `src/device/macos_native/ioreport.rs` (for any chip-specific cadence), `channel_filter.rs` tests if the subscribed set changes.

**Out of scope:** how temperatures are aggregated (a separate decision this data informs), IOReport and SMC read cadence (#414), Intel Macs.

## Implementation Notes

- **Reuse**: `inventory()` and the `include_str!` fixture loading in `energy/tests.rs:25-52`; `EnergyTracker` stays as is unless a chip's timestamps contradict it.
- **Constraints**: the M5 Max fixture and tests keep passing unchanged; `classify_energy_channel` stays the single list of energy channels, because the subscription filter reads it.
- **Edge cases**: counters that publish on every sample, and elements with no timestamp (which run on the observation-clock fallback by design), must both read steady power at `--interval 1`; a chip with `GPU Energy` and per-die GPU channels counts GPU once; a rail channel found only in the inventory must be classified before its cadence can be checked.

## Acceptance Criteria

- [ ] Fixtures and passing classification tests for the M1 Ultra and every other chip tested.
- [ ] Diagnostic output (energy inventory, cadence, SMC temperature key inventory) attached to the PR for each chip.
- [ ] The #410 load check passes on each chip tested, or the deviation is fixed with a test.
- [ ] The `DIE__` decision is implemented, or documented in the `energy.rs` docs with the inventory that justifies it.
- [ ] Chips not tested are listed explicitly in the PR.
- [ ] Any classifier change reaches the real path: the subscription filter and `EnergyTracker` use it, so `all-smi local` and `all-smi api` report the corrected rails.

## Verification

```bash
cargo test --lib ioreport_energy_diagnostics -- --ignored --nocapture # on each chip, idle and under load
cargo test --lib device::macos_native
cargo fmt --check && cargo clippy --all-targets
cargo build --release
yes > /dev/null & yes > /dev/null & yes > /dev/null & yes > /dev/null &
./target/release/all-smi api --port 9090 --interval 1 &
for i in $(seq 20); do curl -s localhost:9090/metrics | grep -E '^all_smi_(cpu|gpu)_power_consumption_watts|^all_smi_ane_power_watts'; sleep 1; done
kill %1 %2 %3 %4 %5
```

Pass: CPU power stays in a steady band with no 0 W sample and matches the CPU rail the diagnostic prints under the same load.

## Related

- #410 / PR #412: the classification rules and publication timing under test.
- #414: IOReport and SMC read cadence on macOS.
- #167: M5 Pro/Max MCPU channels.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/device/macos_native/ioreport.rs and run the ignored ioreport_energy_diagnostics test on the available Apple Silicon hardware, capturing inventories and cadence under idle and load. Compare results with classify_energy_channel, sum_rails, channel_filter.rs, and energy/tests.rs; done means fixtures and classification tests cover tested chips, diagnostics include SMC temperature-key inventory, and any DIE-prefixed handling is implemented or documented without double counting.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability-sre, operating-systems, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.