mlco2 / mlco2/codecarbon

Flaky: tracker flush and logging-output tests fail intermittently on master

Open
#1,371 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
323
Avg merge
1d 12h
Merged PRs (30d)
12

Description

Three tests fail intermittently when the whole suite runs, across two files:

  • tests/test_emissions_tracker_flush.py::test_carbon_tracker_offline_flush
  • tests/test_logging_output.py::test_carbon_tracker_offline_logging_output

The failures are row counts (0 == 1, 2 == 3), so an expected measurement did not land before the assertion. Each test passes in isolation, and it reproduces on master, not just on feature branches.

The most likely cause is how CI runs the suite. .github/workflows/test-package.yml:36 is:

uv run codecarbon monitor -- uv run task test-coverage

so a live tracker wraps the entire test run. It holds /tmp/.codecarbon.lock and writes its own emissions.csv in the repo root for as long as the tests execute, while the tests under it are starting and stopping their own trackers. That is a real interaction and it is not present when you run a single test locally.

Two smaller things that make it worse:

  • tests/test_logging_output.py builds three trackers with no output_dir and no output_file (lines 48-52, 64-69, and the @track_emissions decorator at 81-85), so they all default to emissions.csv in the current working directory. Nothing cleans that up.
  • Both files use a fixed filename in tempfile.gettempdir() (emissions-test-TestCarbonTrackerFlush.csv, emissions-test-TestCarbonLoggingOutput.log) with no per-run suffix, and setUp/tearDown both delete it.

PeriodicScheduler chaining a threading.Timer per tick may also contribute, since a tracker a test does not stop keeps firing into the next test and tests/conftest.py only clears the hardware probe caches. Unverified. The re-arming timer goes away with #1339 / #1324, so it is worth re-measuring after one of those lands.

Likely fixes: give the logging test an explicit output_dir, make the temp filenames unique per process, and stop asserting on wall-clock-dependent row counts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .github/workflows/test-package.yml:36 and reproduce the full suite using the wrapped test command, then inspect the two failing tests in tests/test_emissions_tracker_flush.py and tests/test_logging_output.py. Compare their tracker lifecycles, fixed temporary filenames, default emissions.csv usage, and the timer behavior described in the issue; done means the full suite passes reliably without wall-clock-dependent row-count failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.