mlco2 / mlco2/codecarbon

GPUs without an NVML energy counter report 0 kWh

Open
#1,316 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
323
Avg merge
1d 12h
Merged PRs (30d)
12

Description

Problem

On GPUs where NVML does not implement nvmlDeviceGetTotalEnergyConsumption (pre-Volta cards such as the GTX 1080, Tesla P100 or K80, and many virtualised/vGPU setups), CodeCarbon reports gpu_energy = 0.0 and gpu_power = 0.0 for the entire run. The GPU is still detected and listed in the metadata, so a GPU-bound training run silently reports only its CPU + RAM consumption. The same applies to AMD devices whose amdsmi payload carries no energy accumulator.

Reproduction

Run any tracker on such a GPU, or simulate it by making pynvml.nvmlDeviceGetTotalEnergyConsumption raise NVMLError: AllGPUDevices().get_delta(Time.from_seconds(3600)) returns power_usage = 0 W and delta_energy_consumption = 0 kWh on every device, even though nvmlDeviceGetPowerUsage reports a non-zero draw.

Root cause

NvidiaGPUDevice._get_total_energy_consumption returns None when NVML raises (codecarbon/core/gpu_nvidia.py:55). GPUDevice._get_energy_kwh then returns self.last_energy unchanged (codecarbon/core/gpu_device.py:44), so in GPUDevice.delta (codecarbon/core/gpu_device.py:51) energy == last_energy, giving energy_delta = 0 and Power.from_energies_and_delay(...) = 0 W. In other words "no new reading" is treated as "no energy consumed". Unlike CPU.measure_power_and_energy, the GPU path has no power-integration fallback, even though _get_power_usage() works on every card.

Expected vs actual

Expected: when the cumulative counter is unavailable, integrate the instantaneous power draw over the measurement interval (the approximation the CPU path already uses) and warn once per device.

Actual: zero energy and zero power for the whole run, with only a per-sample logger.warning("Failed to retrieve gpu total energy consumption") that never says the resulting number is zero.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with codecarbon/core/gpu_nvidia.py:55 and codecarbon/core/gpu_device.py:44-51, then trace AllGPUDevices().get_delta(Time.from_seconds(3600)) with the NVML energy call failing. Use the working instantaneous power reading as the fallback for unavailable cumulative energy, cover the analogous AMD case described in the issue, and ensure the device warning is emitted once while the reported delta and power are non-zero.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.