RAPL counter wrap yields negative energy when max_energy_range_uj is unreadable
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 323
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 12
Description
When a RAPL counter wraps around and max_energy_range_uj could not be read, CodeCarbon records a large negative energy delta instead of skipping the sample. That value is summed straight into the CPU and total energy, so long-running sessions can report totals that go down over time.
Reproduction
Point a RAPLFile at an energy file whose max_path does not exist (this is what happens when only energy_uj is readable, e.g. some container/VM RAPL passthroughs, MMIO domains, or a chmod scoped to energy_uj):
write(energy_uj, "4000000000"); f = RAPLFile(...); f.start()
write(energy_uj, "10000"); f.delta(Time.from_seconds(10))
assert f.energy_delta.kWh < 0 # passes today, ~ -1.11e-3 kWh
Root cause
codecarbon/core/rapl.py:45 sets max_energy_reading = 0 on any failure to read max_energy_range_uj, with a comment claiming wrap detection is then "effectively disabled". It is not — only the correction is disabled. codecarbon/core/rapl.py:75-83 still takes the wrap branch, adds 0, and computes energy_delta = energy - last_energy, a large negative number.
codecarbon/core/cpu.py:597 (_build_readable_domains) gates domain acceptance on energy_uj only, so a domain with an unreadable max file is happily monitored. The negative delta then flows through codecarbon/core/cpu.py:847 and codecarbon/external/hardware.py:381-393 into the tracker totals with no sign check.
The same negative delta is produced by the transient-read fallback at codecarbon/core/rapl.py:65, which returns 0 on a read error and therefore trips the wrap branch on the next successful read.
The reported power still looks plausible, because Power.from_energies_and_delay takes abs() (codecarbon/core/units.py:139), which is what makes this hard to spot.
Expected vs actual
- Expected: an uncorrectable backwards counter is dropped (zero delta) with a warning, as
codecarbon/core/windows_emi.py:567-574already does for EMI. - Actual: up to -1.19e-3 kWh subtracted from the totals per wrap (every ~hour at 1 W for a 2^32 uJ package domain, minutes under load).
_update_emissionsskips negative deltas (codecarbon/emissions_tracker.py:957-976), so_last_energy_coveredalso permanently desynchronises from_total_energy.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in codecarbon/core/rapl.py at the max_energy_reading and transient-read paths, then trace callers in cpu.py and hardware.py; compare the existing handling in codecarbon/core/windows_emi.py. Done when the supplied reproduction no longer produces a negative energy_delta and an uncorrectable counter cannot reduce tracker totals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100