ESMCI / ESMCI/cime

perf_check_for_memory_leak reports nonsensical "days" value in memleak message

Open
#5,007 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
174
Forks
225
Avg merge
1d 16h
Merged PRs (30d)
14

Description

The memory-leak check in CIME/SystemTests/system_tests_common.py reports the elapsed time in "days" using arithmetic that is not actually days. For any run longer than a few days, the reported number is off by orders of magnitude and confusing to users.

### Example

A 5-year SMS_Ly5 ELM test produced this TestStatus.log entry:

memleak detected, memory went from 1499.740000 to 1578.320000 in 41129 days

The test only ran for 5 model years (~1825 days), not 41,129.

### Root cause

perf_get_memory_list (in CIME/baselines/performance.py) parses cpl.log lines of the form:

`... model date = 00051231 ... memory = 1578.320 MB ... highwater ...`

using the regex

`meminfo = re.compile(r".*model date =\s+(\w+).*memory =\s+(\d+\.?\d+).*highwater")`

and stores the first capture group as float(m.group(1)). That value is the model date encoded as YYYYMMDD (e.g. 51231.0), not a day count.

Then in perf_check_for_memory_leak (CIME/SystemTests/system_tests_common.py, around line 1109):
```
# last day - second day, skip first day, can be too low while initializing
elapsed_days = int(memlist[-1][0]) - int(memlist[1][0])
```

This subtracts the two encoded stamps directly. For the run above, the second sample was model date = 00010102 and the last sample was model date = 00051231, so:

51231 - 10102 = 41129

which is exactly the "41129 days" printed in the message. The result mixes years (×10000), months (×100), and day-of-month, so it grows by ~10000 per model year plus ~100 per model month and is not a meaningful count of days.

### Impact

- Cosmetic / user-facing only. The pass/fail decision itself is unaffected — that uses finalmem/originalmem and the tolerance ratio, both of which are correct.
- The MEMLEAK TestStatus.log message is misleading, and can send developers looking for a bug in the wrong place ("why does it think my 5-year run went for 100+ years?").
- Any downstream tooling that parses this comment and interprets the number as days will be wrong for any run longer than a few days.

### Suggested fix

Decode the YYYYMMDD stamp back into (year, month, day) before differencing, then report the elapsed time in years/months/days (dropping zero components), e.g.:

memleak detected, memory went from 1499.740000 to 1578.320000 in 4 years, 11 months, 29 days

The same message should be emitted by any component-specific config.perf_check_for_memory_leak override so the wording stays consistent.

Contributor guide

Open the contributing guide

Research direction

Start with perf_get_memory_list in CIME/baselines/performance.py and perf_check_for_memory_leak around line 1109 in CIME/SystemTests/system_tests_common.py. Trace how the YYYYMMDD value is stored and how component-specific config.perf_check_for_memory_leak overrides emit the message. Done means the log reports a meaningful years/months/days duration consistently while the existing memory-leak pass/fail calculation remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.