mlco2 / mlco2/codecarbon

CPU power samples are silently dropped by a race in CPU.total_power()

Open
#1,315 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
323
Avg merge
1d 12h
Merged PRs (30d)
12

Description

In cpu_load mode (and any mode that averages sampled power), CPU._power_history is written by one scheduler thread and drained by another with no synchronisation, so a fraction of the collected samples is discarded before it is averaged. The reported cpu_power is therefore computed from an incomplete, and systematically biased, sample set.

Root cause

Two independent PeriodicScheduler instances are started for every tracker (codecarbon/emissions_tracker.py:345-352), each running on its own threading.Timer thread:

  • the 1 Hz monitor thread calls CPU.monitor_power(), which appends to _power_history (codecarbon/external/hardware.py:433-435);
  • the measure_power_secs thread calls CPU.total_power() (codecarbon/external/hardware.py:395-403):
def total_power(self) -> Power:
    self._power_history.append(self._get_power_from_cpus())
    power_history_in_W = [power.W for power in self._power_history]   # read
    self._power_history = []                                          # rebind
    ...

list.append is atomic under the GIL, but this read-then-rebind pair is not. Any sample appended by the monitor thread between the comprehension and the rebinding lands in the old list object and is then thrown away. The window is not negligible: _get_power_from_cpus() may shell out to a subprocess (e.g. IntelPowerGadget.get_cpu_details, codecarbon/core/cpu.py), during which the monitor thread can fire several times.

Secondary: the if not power_history_in_W: branch on hardware.py:399 is unreachable — line 396 unconditionally appends a sample first, so the list is never empty.

Reproduction

Deterministic version, no timing dependence:

cpu = CPU.from_utils(None, MODE_CPU_LOAD, "some model", 100)
# make _get_power_from_cpus slow (e.g. sleep 100 ms)
# thread A: call cpu.monitor_power() in a loop
# thread B: call cpu.total_power()
# assert samples consumed by B == samples produced by A

The counts do not match; the samples produced during B's slow measurement are lost.

Expected vs actual

Expected: every sample appended by the monitor thread is included in exactly one average.
Actual: samples appended during the swap window are dropped. On a bursty workload the dropped samples are exactly those taken while a slow measurement is in flight, so the error is small (a few percent) but systematic and invisible to the user.

Suggested fix

Guard the swap with a threading.Lock held only for the O(1) rebinding, keeping _get_power_from_cpus() outside the lock so a slow backend never blocks the monitor thread, and drop the dead branch.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in codecarbon/external/hardware.py with CPU.total_power() and CPU.monitor_power(), then inspect the two scheduler call sites in codecarbon/emissions_tracker.py:345-352. Reproduce the deterministic concurrent scenario described in the issue and verify that samples produced during measurement are consumed exactly once. Done means the swap is synchronized without blocking slow power measurement and the unreachable branch is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.