GPUs without an NVML energy counter report 0 kWh
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 323
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 12
Description
Problem
On GPUs where NVML does not implement nvmlDeviceGetTotalEnergyConsumption (pre-Volta cards such as the GTX 1080, Tesla P100 or K80, and many virtualised/vGPU setups), CodeCarbon reports gpu_energy = 0.0 and gpu_power = 0.0 for the entire run. The GPU is still detected and listed in the metadata, so a GPU-bound training run silently reports only its CPU + RAM consumption. The same applies to AMD devices whose amdsmi payload carries no energy accumulator.
Reproduction
Run any tracker on such a GPU, or simulate it by making pynvml.nvmlDeviceGetTotalEnergyConsumption raise NVMLError: AllGPUDevices().get_delta(Time.from_seconds(3600)) returns power_usage = 0 W and delta_energy_consumption = 0 kWh on every device, even though nvmlDeviceGetPowerUsage reports a non-zero draw.
Root cause
NvidiaGPUDevice._get_total_energy_consumption returns None when NVML raises (codecarbon/core/gpu_nvidia.py:55). GPUDevice._get_energy_kwh then returns self.last_energy unchanged (codecarbon/core/gpu_device.py:44), so in GPUDevice.delta (codecarbon/core/gpu_device.py:51) energy == last_energy, giving energy_delta = 0 and Power.from_energies_and_delay(...) = 0 W. In other words "no new reading" is treated as "no energy consumed". Unlike CPU.measure_power_and_energy, the GPU path has no power-integration fallback, even though _get_power_usage() works on every card.
Expected vs actual
Expected: when the cumulative counter is unavailable, integrate the instantaneous power draw over the measurement interval (the approximation the CPU path already uses) and warn once per device.
Actual: zero energy and zero power for the whole run, with only a per-sample logger.warning("Failed to retrieve gpu total energy consumption") that never says the resulting number is zero.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with codecarbon/core/gpu_nvidia.py:55 and codecarbon/core/gpu_device.py:44-51, then trace AllGPUDevices().get_delta(Time.from_seconds(3600)) with the NVML energy call failing. Use the working instantaneous power reading as the fallback for unavailable cumulative energy, cover the analogous AMD case described in the issue, and ensure the device warning is emitted once while the reported delta and power are non-zero.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100