mlco2 / mlco2/codecarbon

RAPL counter wrap yields negative energy when max_energy_range_uj is unreadable

Open
#1,309 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
323
Avg merge
1d 12h
Merged PRs (30d)
12

Description

When a RAPL counter wraps around and max_energy_range_uj could not be read, CodeCarbon records a large negative energy delta instead of skipping the sample. That value is summed straight into the CPU and total energy, so long-running sessions can report totals that go down over time.

Reproduction

Point a RAPLFile at an energy file whose max_path does not exist (this is what happens when only energy_uj is readable, e.g. some container/VM RAPL passthroughs, MMIO domains, or a chmod scoped to energy_uj):

write(energy_uj, "4000000000"); f = RAPLFile(...); f.start()
write(energy_uj, "10000");      f.delta(Time.from_seconds(10))
assert f.energy_delta.kWh < 0   # passes today, ~ -1.11e-3 kWh

Root cause

codecarbon/core/rapl.py:45 sets max_energy_reading = 0 on any failure to read max_energy_range_uj, with a comment claiming wrap detection is then "effectively disabled". It is not — only the correction is disabled. codecarbon/core/rapl.py:75-83 still takes the wrap branch, adds 0, and computes energy_delta = energy - last_energy, a large negative number.

codecarbon/core/cpu.py:597 (_build_readable_domains) gates domain acceptance on energy_uj only, so a domain with an unreadable max file is happily monitored. The negative delta then flows through codecarbon/core/cpu.py:847 and codecarbon/external/hardware.py:381-393 into the tracker totals with no sign check.

The same negative delta is produced by the transient-read fallback at codecarbon/core/rapl.py:65, which returns 0 on a read error and therefore trips the wrap branch on the next successful read.

The reported power still looks plausible, because Power.from_energies_and_delay takes abs() (codecarbon/core/units.py:139), which is what makes this hard to spot.

Expected vs actual

  • Expected: an uncorrectable backwards counter is dropped (zero delta) with a warning, as codecarbon/core/windows_emi.py:567-574 already does for EMI.
  • Actual: up to -1.19e-3 kWh subtracted from the totals per wrap (every ~hour at 1 W for a 2^32 uJ package domain, minutes under load). _update_emissions skips negative deltas (codecarbon/emissions_tracker.py:957-976), so _last_energy_covered also permanently desynchronises from _total_energy.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in codecarbon/core/rapl.py at the max_energy_reading and transient-read paths, then trace callers in cpu.py and hardware.py; compare the existing handling in codecarbon/core/windows_emi.py. Done when the supplied reproduction no longer produces a negative energy_delta and an uncorrectable counter cannot reduce tracker totals.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.