mlco2 / mlco2/codecarbon

Support AMD MIxxx double-die

Open
#1,097 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.9k
Forks
323
Avg merge
1d 12h
Merged PRs (30d)
12

Description

Description:

The AMD Instinct MI250 accelerator card contains two Graphics Compute Dies (GCDs) per physical card. However, when monitoring energy consumption (e.g., via rocm-smi or tools like CodeCarbon), only one GCD reports power usage, while the other shows zero values. This is problematic for accurate energy accounting, especially in HPC/SLURM environments where jobs may be allocated a single GCD.
Expected Behavior:
Both GCDs on the same MI250 card should report their individual power consumption, or the total card power should be clearly attributed to the active GCD(s).

Current Behavior:

Only one GCD provides non-zero power readings.
The second GCD always reports 0W, even when under load.
This leads to underestimated energy measurements and complicates per-job accounting.

Steps to Reproduce:

Allocate a single GCD on an MI250 card via SLURM (e.g., --gres=gpu:1).
Run a workload on the allocated GCD.
Use rocm-smi --showpower or similar tools to monitor energy.
Observe that the second GCD (on the same card) reports 0W, despite the card’s total power draw.

Impact:

Inaccurate energy tracking for jobs sharing a card.
Difficulty distinguishing per-GCD power usage.
Tools like CodeCarbon may misreport energy if they rely on per-GCD metrics.
Suggested Fix:

Provide a way to query total card power (sum of both GCDs) when monitoring a single GCD.
Alternatively, expose power readings for both GCDs, even if only one is allocated to a job.

Context:

This issue affects users in SLURM/HPC environments where fine-grained energy monitoring is critical for carbon footprint tracking and resource management.

Additional Notes:

The MI300 series may have similar behavior; clarification would be helpful.
Workarounds (e.g., manually summing GCDs) are error-prone and not scalable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file, test, or entry point is named in the issue. Start by reproducing the MI250 behavior with rocm-smi under a single-GCD SLURM allocation, then trace how CodeCarbon reads accelerator power; done means both GCD usage or clearly attributed total card power is reported and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
hpc, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.