ESCOMP / ESCOMP/CTSM

Try to speed up interpMonthlyVeg

Open
#731 3 comments 0 reactions 0 assignees View on GitHub
enhancement performance priority: low
Dominant language
Fortran
Stars
352
Forks
361
Avg merge
2d 21h
Merged PRs (30d)
7

Description

In runs of the NWP configuration with NLDAS grid performed by @barlage , he found that interpMonthlyVeg was taking more than 50% of the total run time. I have tried to reproduce his runs using ctsm1.0.dev042 and `./create_newcase --case nwp_nldas_0523a --compset I2000Ctsm50NwpSpNldasRsGs --res nldas2_rnldas2_mnldas2`. I ran a few one or two-month runs, with `./xmlchange TIMER_DETAIL=10,TIMER_LEVEL=20` and the following code changes to add extra timers:

```diff
diff --git a/src/biogeochem/SatellitePhenologyMod.F90 b/src/biogeochem/SatellitePhenologyMod.F90
index 7f5f376..64d2166 100644
--- a/src/biogeochem/SatellitePhenologyMod.F90
+++ b/src/biogeochem/SatellitePhenologyMod.F90
@@ -597,6 +597,8 @@ contains

! Determine necessary indices

+ call t_startf('rmv allocate')
+
allocate(&
mlai(bounds%begg:bounds%endg,0:maxveg), &
msai(bounds%begg:bounds%endg,0:maxveg), &
@@ -608,13 +610,16 @@ contains
call endrun(msg=errMsg(sourcefile, __LINE__))
end if

+ call t_stopf('rmv allocate')
! ----------------------------------------------------------------------
! Open monthly vegetation file
! Read data and convert from gridcell to patch data
! ----------------------------------------------------------------------

+ call t_startf('rmv open')
call getfil(fveg, locfn, 0)
call ncd_pio_openfile (ncid, trim(locfn), 0)
+ call t_stopf('rmv open')

if (single_column) then
call shr_scam_getCloseLatLon (ncid, scmlat, scmlon, closelat, closelon,&
@@ -623,6 +628,7 @@ contains

do k=1,2 !loop over months and read vegetated data

+ call t_startf('rmv ncd_io')
call ncd_io(ncid=ncid, varname='MONTHLY_LAI', flag='read', data=mlai, dim1name=grlnd, &
nt=months(k), readvar=readvar)
if (.not. readvar) call endrun(msg=' ERROR: MONTHLY_LAI NOT on fveg file'//errMsg(sourcefile, __LINE__))
@@ -638,11 +644,13 @@ contains
call ncd_io(ncid=ncid, varname='MONTHLY_HEIGHT_BOT', flag='read', data=mhgtb, dim1name=grlnd, &
nt=months(k), readvar=readvar)
if (.not. readvar) call endrun(msg=' ERROR: MONTHLY_HEIGHT_TOP NOT on fveg file'//errMsg(sourcefile, __LINE__))
+ call t_stopf('rmv ncd_io')

! Only vegetated patches have nonzero values
! Assign lai/sai/hgtt/hgtb to the top [maxsoil_patches] patches
! as determined in subroutine surfrd

+ call t_startf('rmv loop')
do p = bounds%begp,bounds%endp
g =patch%gridcell(p)
if (patch%itype(p) /= noveg) then ! vegetated pft
@@ -661,6 +669,7 @@ contains
mhvb2t(p,k) = 0._r8
end if
end do ! end of loop over patches
+ call t_stopf('rmv loop')

end do ! end of loop over months
```

I wasn't able to reproduce quite the extreme results he found: for me, interpMonthlyVeg took roughly 10% of the total lnd run time. I wonder if cheyenne was having glade issues at the time Mike ran? But this is still worth investigating to see if we can reduce this time.

Through my extra timers, I found that nearly all of the time is being spent in the ncd_io calls. I was suspicious of the loop that sets values, because this is written in an inefficient way (there shouldn't be a need for the double nesting of the loop), but the time spent in this loop is negligible compared with the i/o time.

Two initial ideas for getting some speed-up here, which we could address through experimentation and/or consultation with Jim Edwards:

- Would it be more efficient if we arranged the dimensions of these variables as `time, lsmlat, lsmlon, lsmpft` rather than `time, lsmpft, lsmlat, lsmlon`?

- Can we tweak some pio settings to improve the performance of these reads?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.