Increased run time for a benchmark ne120 F case on Perlmutter

We have been testing a benchmark ne120 F case on Perlmutter and Frontier.

Case settings

Compset: F2010
Resolution: ne120pg2_r05_oECv3

STOP_N=1
STOP_OPTION=ndays

Write frequency: every 1 hour (nhtfrq = -1)

Optional: REST_OPTION=none (disables writing out restart files)

PE layout

<entry id="MAX_TASKS_PER_NODE" value="64">
<entry id="MAX_MPITASKS_PER_NODE" value="64">

<entry id="NTASKS">
  <values>
    <value component="ATM">21600</value>
    <value component="CPL">21600</value>
    <value component="OCN">21600</value>
    <value component="WAV">21600</value>
    <value component="GLC">21600</value>
    <value component="ICE">5400</value>
    <value component="ROF">21600</value>
    <value component="LND">21600</value>
    <value component="ESP">1</value>
    <value compclass="IAC">1</value>
  </values>
</entry>

Performance degradation observed

• July 2023: case.run time was < 5 minutes
• December 2024: case.run time increased to ~10 minutes
• February 2025: case.run time further increased to ~18 minutes

Additionally, "Init time" (shown in the PACE link) increased to >16 minutes in the February 2025 run.
IO stats also indicate that tot_rtime has significantly increased compared to previous runs.

Performance logs

[2023-07-25 case run] (REST_OPTION="none")
PACE Link: https://pace.ornl.gov/exp-details/154317
Run_time: 47.217 sec
Init time: 189.878 sec

case.run time: about 5 mins
2023-07-25 14:09:54: case.run starting 12314437
...
2023-07-25 14:14:24: case.run success 12314437

"OverallIOStatistics":
"avg_wtput(MB/s)" : 14341.042886
"avg_rtput(MB/s)" : 72076.466363
"tot_wb(bytes)" : 163062403693
"tot_rb(bytes)" : 11457075418660
"tot_wtime(s)" : 10.843593
"tot_rtime(s)" : 151.593427
"tot_time(s)" : 168.650298

[2024-12-04 case run]
PACE Link: https://pace.ornl.gov/exp-details/202812
Run_time: 107.786 sec
Init time: 462.357 sec

case.run time: about 10 mins
2024-12-04 11:34:25: case.run starting 33534166
...
2024-12-04 11:44:38: case.run success 33534166

"OverallIOStatistics":
"avg_wtput(MB/s)" : 11581.018884
"avg_rtput(MB/s)" : 56658.159505
"tot_wb(bytes)" : 437643466533
"tot_rb(bytes)" : 11654545209348
"tot_wtime(s)" : 36.039086
"tot_rtime(s)" : 196.170164
"tot_time(s)" : 260.371857

[2024-12-12 case run]
PACE Link: https://pace.ornl.gov/exp-details/203176
Run_time: 107.577 sec
Init time: 576.172 sec

case.run time: about 12 mins
2024-12-12 02:52:27: case.run starting 33812266
...
2024-12-12 03:04:05: case.run success 33812266

"OverallIOStatistics":
"avg_wtput(MB/s)" : 10940.771565
"avg_rtput(MB/s)" : 51752.134239
"tot_wb(bytes)" : 437643466533
"tot_rb(bytes)" : 11654545209348
"tot_wtime(s)" : 38.148071
"tot_rtime(s)" : 214.766803
"tot_time(s)" : 281.167514

[2025-02-12 case run] (REST_OPTION="none")
PACE Link: https://pace.ornl.gov/exp-details/209893
Run_time: 91.013 sec
Init time: 974.670 sec

case.run time: about 18 mins
2025-02-12 08:15:21: case.run starting 35762678
...
2025-02-12 08:33:21: case.run success 35762678

"OverallIOStatistics":
"avg_wtput(MB/s)" : 8791.979204
"avg_rtput(MB/s)" : 24544.182600
"tot_wb(bytes)" : 318305933857
"tot_rb(bytes)" : 11654545209348
"tot_wtime(s)" : 34.526946
"tot_rtime(s)" : 452.842151
"tot_time(s)" : 481.267834

Summary of concerns

• Significant increase in case.run time: From <5 min (July 2023) to ~18 min (February 2025).
• Much longer Init time: From 189 sec (July 2023) to 974 sec (February 2025).