EAMxx: Performance issue on Frontier

There seems to be a potential performance issue running a recent version of EAMxx at large-nodes on Frontier. My first setup is based on uniform ne256pg2 grid and 'F2010-SCREAMv1' compset designed to perform a year-long simulation. On 256-nodes with 12h wall-clock submission, the simulation advanced about 8-months. I expected the simulation would advance a couple months further based on my earlier benchmarks. However, when I plotted the difference in hourly file creation timestamps, I noticed there is a slowdown after about 3-months (~ 3x745) and gradually becoming slower later. So, normally it takes about 3-4 seconds for the setup to generate 2D hourly files, but later it shows 10s or slower to generate such files (plot 1). The issue showed up again for the next restart submission with hourly file generation taking 30s or more (plot 2). I also have daily averaged 2D outputs along with 30-days restart outputs which correspond to some of the peaks.



My second setup is based on an high-res RRM grid and again utilizing 256-nodes. For this setup, slowdown is more significant (from 10s per file to ~150s and more) as it is shown in plots 3 and 4. Plot 4 is generated from a new re-submission of the same case. In both RRM runs, slowdown starts right at the same file - `output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-28800.nc` (about 1-h into the simulation). Following shows a sample of file listing to highlight differences in file creation timestamps.

```
...
2025-05-29_19:44:23 output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-18000.nc
2025-05-29_19:44:33 output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-21600.nc
2025-05-29_19:44:42 output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-25200.nc
2025-05-29_19:46:07 output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-28800.nc
2025-05-29_19:48:20 output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-32400.nc
2025-05-29_19:50:31 output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-36000.nc
2025-05-29_19:52:46 output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-39600.nc
...
```



Based on discussions in frontier-e3sm channel, it is more likely that this issue is code related, thus we are documenting the case here for further comments and investigation.

Reproduce the slowdown using the two described 256-node Frontier setups: the ne256pg2 F2010-SCREAMv1 case and the high-resolution RRM case. Start by examining hourly file-creation intervals around output.scream.SurfVars1.INSTANT.nhours_x1.2017-03-04-28800.nc and compare them with the supplied plots and listings. Done means identifying the cause of the increasing output time and confirming the performance behavior is corrected.

Written by the indexing model from the issue text.