Issues with Environment Variable "NUMEXPR_MAX_THREADS" // Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz

I am using CodeCarbon to profile High Energy Physics software, and I encountered an issue when running it on an Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz with cpu_count = 96. I receive the following error:

Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64)[codecarbon INFO @ 10:16:11] [setup] RAM Tracking...

After this, tracking begins as expected. However, the reported CPU power consumption is approximately 95 W. In contrast, when I request 64 cores, the error disappears, and the CPU power consumption increases to approximately 150 W. This discrepancy suggests that CodeCarbon may be struggling to obtain correct RAPL readings when using 96 cores.

I am working in a High-Performance Computing Cluster (HPCC) with an operating system: Linux-5.14.0-503.22.1.el9_5.x86_64-x86_64-with-glibc2.34.

Below is a copy of the output for 96 Cores requested:

Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64)[codecarbon INFO @ 10:16:11] [setup] RAM Tracking...
[codecarbon INFO @ 10:16:11] [setup] GPU Tracking...
[codecarbon INFO @ 10:16:11] No GPU found.
[codecarbon INFO @ 10:16:11] [setup] CPU Tracking...
[codecarbon INFO @ 10:16:11] Tracking Intel CPU via RAPL interface
[codecarbon INFO @ 10:16:13] >>> Tracker's metadata:
[codecarbon INFO @ 10:16:13] Platform system: Linux-5.14.0-503.23.1.el9_5.x86_64-x86_64-with-glibc2.34
[codecarbon INFO @ 10:16:13] Python version: 3.9.12
[codecarbon INFO @ 10:16:13] CodeCarbon version: 2.7.1
[codecarbon INFO @ 10:16:13] Available RAM : 376.013 GB
[codecarbon INFO @ 10:16:13] CPU count: 96
[codecarbon INFO @ 10:16:13] CPU model: Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz
[codecarbon INFO @ 10:16:13] GPU count: None
[codecarbon INFO @ 10:16:13] GPU model: None
[codecarbon INFO @ 10:16:17] Saving emissions data to file /scratch/condor_pool/condor/dir_3450188/emissions.csv
[codecarbon INFO @ 10:16:32] Energy consumed for RAM : 0.000588 kWh. RAM Power : 141.00501823425293 W
[codecarbon INFO @ 10:16:32] Energy consumed for all CPUs : 0.000319 kWh. Total CPU Power : 76.6423050726188 W
[codecarbon INFO @ 10:16:32] 0.000907 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:16:47] Energy consumed for RAM : 0.001175 kWh. RAM Power : 141.00501823425293 W
[codecarbon INFO @ 10:16:47] Energy consumed for all CPUs : 0.000642 kWh. Total CPU Power : 77.5457339125873 W
[codecarbon INFO @ 10:16:47] 0.001818 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:17:02] Energy consumed for RAM : 0.001762 kWh. RAM Power : 141.00501823425293 W
[codecarbon INFO @ 10:17:02] Energy consumed for all CPUs : 0.000975 kWh. Total CPU Power : 79.89638462293308 W
[codecarbon INFO @ 10:17:02] 0.002738 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:17:17] Energy consumed for RAM : 0.002350 kWh. RAM Power : 141.00501823425293 W
[codecarbon INFO @ 10:17:17] Energy consumed for all CPUs : 0.001307 kWh. Total CPU Power : 79.65950132244772 W
[codecarbon INFO @ 10:17:17] 0.003657 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:17:32] Energy consumed for RAM : 0.002937 kWh. RAM Power : 141.00501823425293 W
[codecarbon INFO @ 10:17:32] Energy consumed for all CPUs : 0.001696 kWh. Total CPU Power : 93.23843614582798 W
[codecarbon INFO @ 10:17:32] 0.004633 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:17:47] Energy consumed for RAM : 0.003525 kWh. RAM Power : 141.00501823425293 W
[codecarbon INFO @ 10:17:47] Energy consumed for all CPUs : 0.002089 kWh. Total CPU Power : 94.39081254152961 W
[codecarbon INFO @ 10:17:47] 0.005614 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:18:02] Energy consumed for RAM : 0.004112 kWh. RAM Power : 141.00501823425293 W
[codecarbon INFO @ 10:18:02] Energy consumed for all CPUs : 0.002486 kWh. Total CPU Power : 95.19929476561141 W
[codecarbon INFO @ 10:18:02] 0.006598 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:18:17] Energy consumed for RAM : 0.004700 kWh. RAM Power : 141.00501823425293 W
[codecarbon INFO @ 10:18:17] Energy consumed for all CPUs : 0.002879 kWh. Total CPU Power : 94.48069665464713 W
...

Below is a copy of the output for 64 cores requested:

[codecarbon INFO @ 09:58:22] [setup] RAM Tracking...
[codecarbon INFO @ 09:58:22] [setup] GPU Tracking...
[codecarbon INFO @ 09:58:22] No GPU found.
[codecarbon INFO @ 09:58:22] [setup] CPU Tracking...
[codecarbon INFO @ 09:58:22] Tracking Intel CPU via RAPL interface
[codecarbon INFO @ 09:58:24] >>> Tracker's metadata:
[codecarbon INFO @ 09:58:24] Platform system: Linux-5.14.0-503.23.1.el9_5.x86_64-x86_64-with-glibc2.34
[codecarbon INFO @ 09:58:24] Python version: 3.9.12
[codecarbon INFO @ 09:58:24] CodeCarbon version: 2.7.1
[codecarbon INFO @ 09:58:24] Available RAM : 376.013 GB
[codecarbon INFO @ 09:58:24] CPU count: 96
[codecarbon INFO @ 09:58:24] CPU model: Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz
[codecarbon INFO @ 09:58:24] GPU count: None
[codecarbon INFO @ 09:58:24] GPU model: None
[codecarbon INFO @ 09:58:27] Saving emissions data to file /scratch/condor_pool/condor/dir_4129677/emissions.csv
[codecarbon INFO @ 09:58:42] Energy consumed for RAM : 0.000588 kWh. RAM Power : 141.00501108169556 W
[codecarbon INFO @ 09:58:42] Energy consumed for all CPUs : 0.000692 kWh. Total CPU Power : 165.97216078184402 W
[codecarbon INFO @ 09:58:42] 0.001279 kWh of electricity used since the beginning.
[codecarbon INFO @ 09:58:57] Energy consumed for RAM : 0.001175 kWh. RAM Power : 141.00501108169556 W
[codecarbon INFO @ 09:58:57] Energy consumed for all CPUs : 0.001386 kWh. Total CPU Power : 166.7243149180381 W
[codecarbon INFO @ 09:58:57] 0.002561 kWh of electricity used since the beginning.
[codecarbon INFO @ 09:59:12] Energy consumed for RAM : 0.001762 kWh. RAM Power : 141.00501108169556 W
[codecarbon INFO @ 09:59:12] Energy consumed for all CPUs : 0.002081 kWh. Total CPU Power : 166.7314009239928 W
[codecarbon INFO @ 09:59:12] 0.003844 kWh of electricity used since the beginning.
[codecarbon INFO @ 09:59:27] Energy consumed for RAM : 0.002350 kWh. RAM Power : 141.00501108169556 W
[codecarbon INFO @ 09:59:27] Energy consumed for all CPUs : 0.002776 kWh. Total CPU Power : 166.78664661653738 W
[codecarbon INFO @ 09:59:27] 0.005126 kWh of electricity used since the beginning.
[codecarbon INFO @ 09:59:42] Energy consumed for RAM : 0.002937 kWh. RAM Power : 141.00501108169556 W
[codecarbon INFO @ 09:59:42] Energy consumed for all CPUs : 0.003487 kWh. Total CPU Power : 170.69463290630506 W
[codecarbon INFO @ 09:59:42] 0.006425 kWh of electricity used since the beginning.
[codecarbon INFO @ 09:59:57] Energy consumed for RAM : 0.003525 kWh. RAM Power : 141.00501108169556 W
[codecarbon INFO @ 09:59:57] Energy consumed for all CPUs : 0.004205 kWh. Total CPU Power : 172.23786010192958 W
[codecarbon INFO @ 09:59:57] 0.007730 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:00:12] Energy consumed for RAM : 0.004112 kWh. RAM Power : 141.00501108169556 W
[codecarbon INFO @ 10:00:12] Energy consumed for all CPUs : 0.004930 kWh. Total CPU Power : 173.9534875238764 W
[codecarbon INFO @ 10:00:12] 0.009042 kWh of electricity used since the beginning.
[codecarbon INFO @ 10:00:27] Energy consumed for RAM : 0.004700 kWh. RAM Power : 141.00501108169556 W
[codecarbon INFO @ 10:00:27] Energy consumed for all CPUs : 0.005653 kWh. Total CPU Power : 173.56272248219523 W
...

I attempted to modify "NUMEXPR_MAX_THREADS" inside the Python file using os, but this did not resolve the issue. I also tried setting "NUMEXPR_MAX_THREADS" directly in Bash just before executing the Python script. While this eliminated the error message, the reported CPU power consumption remained at ~95 W, suggesting that CodeCarbon is still not reading RAPL correctly when using 96 cores. Is there something I am missing or misunderstanding?

Start by reproducing the reported behavior with 64 and 96 CPUs, including the NUMEXPR_MAX_THREADS setting. Inspect the CPU Tracking path that reports the Intel RAPL interface and compare its readings in both runs. Done means explaining the discrepancy and correcting the 96-core measurement or documenting the relevant limitation.