[Feature]: Optional informative-only power/energy telemetry tier for results that cannot use PTDaemon

Summary

An opt-in, clearly-labelled "informative only" energy tier that records on-device power telemetry (NVML / RAPL / powermetrics) alongside the existing performance metrics, for the large majority of v2.0 users who can run the benchmark from home but cannot assemble a PTDaemon + power-analyzer setup.

Interface type

Both

Application version

v2.0.0

Operating system

Not OS-specific (telemetry source differs per platform: NVML on Windows/Linux NVIDIA, powermetrics on macOS, RAPL on x86)

Motivation

The v2.0 release makes the point explicitly: "most of our members can generate their own results from home." That is a big change in who can produce a data point — but the energy axis did not come along with it. Certified MLPerf power measurement needs a Yokogawa-class analyzer, SPEC PTDaemon, a second director machine and NTP discipline. That is correct and non-negotiable for a certified number, and I am not proposing to weaken it. The consequence, though, is that a run done from home is silent about energy, which is the one axis a home or small-lab user most often cares about and the one axis where the interesting cross-runtime differences live.

Concretely, why the gap matters: I measured Llama-3.1-8B-Instruct decoding on one RTX 4090 with GPU-package power sampled directly from NVML at 10 Hz, comparing GGUF F16 against GGUF Q4_0 under the same llama.cpp binary — including the exact Q4_0 artifact this repo ships for the NVIDIA llama.cpp CUDA scenario
(data/configs/vendors_default/llm/Llama3.1/NVIDIA_llamacpp-CUDA_GPU.json ->
https://client.mlcommons-storage.org/deps/2.0/scenario_files/models/llama3/8b-instruct/GGML/Llama-3.1-8B-Instruct-Q4_0.gguf).

Provisional result (n=3, one card, one session — caveats below and in the write-up):

arm	mJ/token	tok/s	decode avg power	vs F16
F16	4815	63.8	307 W	—
Q4_0 (ours)	1754	168.6	296 W	-63.6%
Q4_0 (this repo's file)	1795	172.6	310 W	-62.7%

Perplexity 7.3260 -> 7.7364 (+5.6%, wikitext-2-raw-v1, -c 512, 564 chunks).

The mechanism is the part worth surfacing in a benchmark: the three arms draw nearly the same instantaneous power (296-310 W). The entire energy saving comes from throughput, not from a lower power draw. That is exactly the kind of statement a performance-only report cannot make, and it is also why "4-bit" is not a portable energy claim — on the same card, bitsandbytes LLM.int8() through transformers costs +106% energy per token, while llama.cpp Q4_0 saves ~64%. Same weight-only quantization idea, opposite sign, because the kernel differs. A telemetry tier would let MLPerf Client surface that class of difference across the EPs it already enumerates.

Proposed solution or approach

Add an opt-in flag, e.g. --energy-telemetry (GUI: a checkbox), that:

1. Samples on-device power for the duration of each scenario and integrates it to joules:
   • NVIDIA: nvmlDeviceGetPowerUsage
   • AMD: rocm-smi / ADLX equivalent
   • Intel / x86 CPU: RAPL
   • Apple silicon: powermetrics
2. Emits, per scenario, alongside the existing metrics:
   • energy_j, mean_power_w, sample_rate_hz, n_samples
   • derived mJ/token and tokens/J for the generation phase
   • the telemetry source string (e.g. "NVML gpu-package")
3. Marks every such field in the result JSON as informative:
   • "power_measurement_tier": "informative_telemetry"
   • "certified_power_result": false
     and refuses to render them in any view that could be mistaken for a certified power result.

The boundary that must be stated in the schema itself, not just in docs: this is component-level (GPU package / CPU package) power, not whole-system wall AC power. It excludes VRM losses upstream of the sensor, PSU inefficiency, display, storage and the rest of the platform. It is therefore not comparable to a PTDaemon number and must never be aggregated with one.

The scoping decision I would suggest: gate it behind the flag, keep it out of the default result summary, and never allow it into a submission path. It is a diagnostic and a research signal, not a score.

For reference, the sampler and the differencing protocol I used are small and MIT-licensed:
https://github.com/hongping-zh/ecocompute-mlcube
Method write-up and the raw per-run CSV (18 runs, undifferenced process energy, first/mean/peak power, cold-start flags):
https://quantenergy.tech/blog/it-was-never-the-format.html

Alternatives considered

• Running the benchmark and a separate sampler side by side (what I did). It works, but the sampler cannot start and stop inside the benchmark process, so the energy window includes model load and teardown. On an 8B F16 model that overhead is larger than the decode itself. I had to recover a decode-only figure by differencing a 576-token run against a 64-token run, which is an approximation an in-process sampler would make unnecessary.
• Reporting only performance and letting readers infer energy from tok/s. This is exactly what breaks: it assumes power is constant across backends, and while that happened to hold within llama.cpp here, it does not hold across runtimes.
• Waiting for a full PTDaemon setup. That is the right answer for a certified number and the wrong answer for the "run it from home" audience v2.0 just unlocked.

Hardware configuration (optional)

• System type: rented Linux GPU instance
• GPU: NVIDIA GeForce RTX 4090 24 GB, power limit 450 W (stock)
• CUDA 12.x, llama.cpp b10643 (192067b72)

Contact information (optional)

zhanghongping1982@gmail.com

Review data/configs/vendors_default/llm/Llama3.1/NVIDIA_llamacpp-CUDA_GPU.json and trace how scenario metrics are collected and emitted. Assess the proposed --energy-telemetry entry point with NVML, RAPL, powermetrics, and other platform sources. Done means opt-in informative telemetry is recorded with its source and schema boundary, excluded from certified or submission results, and covered by tests.