[Discussion] Adding energy consumption metrics to MLPerf Inference Benchmark

Discussion: Energy Metrics for MLPerf Inference

Context

MLPerf Inference currently reports throughput and latency metrics. As AI sustainability becomes a key concern, standardized energy efficiency metrics would complement existing benchmarks.

Observation

Through systematic benchmarking of quantized LLM inference (NF4, INT8, FP16) across NVIDIA Ada Lovelace and Blackwell architectures, we found that:

1. Quantization's energy impact is non-trivial and model-size dependent
2. For models <3B parameters, NF4 quantization increases energy by 25-56%
3. INT8 mixed-precision adds 17-33% energy overhead vs FP16
4. These trade-offs are not captured by throughput/latency alone

Suggestion

Consider adding optional energy reporting to the MLPerf Inference benchmark:

• Energy per query/token (J)
• Average power draw (W)
• Energy efficiency (tokens/J)

This would enable apples-to-apples energy comparison across hardware and quantization configurations.

Data

• Full benchmark dataset (200+ measurements): Zenodo
• Profiling toolkit: EcoCompute-AI
• Interactive results: https://hongping-zh.github.io/ecocompute-dynamic-eval/

Related

• huggingface/transformers#44407 — Energy efficiency docs (approved)
• huggingface/optimum#2410 — Quantization energy data
• vllm-project/vllm#36440 — Energy metrics feature request

No file, test, or entry point is identified. Start by reviewing the existing MLPerf Inference benchmark metric-reporting flow and the proposed energy measurements, then determine where an agreed measurement protocol would fit. Done requires a maintainer-approved scope and implementation plan for energy per query/token, average power, and energy efficiency.