Try to speed up interpMonthlyVeg
- Dominant language
- Fortran
- Stars
- 352
- Forks
- 361
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 7
Description
In runs of the NWP configuration with NLDAS grid performed by @barlage , he found that interpMonthlyVeg was taking more than 50% of the total run time. I have tried to reproduce his runs using ctsm1.0.dev042 and `./create_newcase --case nwp_nldas_0523a --compset I2000Ctsm50NwpSpNldasRsGs --res nldas2_rnldas2_mnldas2`. I ran a few one or two-month runs, with `./xmlchange TIMER_DETAIL=10,TIMER_LEVEL=20` and the following code changes to add extra timers:
```diff
diff --git a/src/biogeochem/SatellitePhenologyMod.F90 b/src/biogeochem/SatellitePhenologyMod.F90
index 7f5f376..64d2166 100644
--- a/src/biogeochem/SatellitePhenologyMod.F90
+++ b/src/biogeochem/SatellitePhenologyMod.F90
@@ -597,6 +597,8 @@ contains
! Determine necessary indices
+ call t_startf('rmv allocate')
+
allocate(&
mlai(bounds%begg:bounds%endg,0:maxveg), &
msai(bounds%begg:bounds%endg,0:maxveg), &
@@ -608,13 +610,16 @@ contains
call endrun(msg=errMsg(sourcefile, __LINE__))
end if
+ call t_stopf('rmv allocate')
! ----------------------------------------------------------------------
! Open monthly vegetation file
! Read data and convert from gridcell to patch data
! ----------------------------------------------------------------------
+ call t_startf('rmv open')
call getfil(fveg, locfn, 0)
call ncd_pio_openfile (ncid, trim(locfn), 0)
+ call t_stopf('rmv open')
if (single_column) then
call shr_scam_getCloseLatLon (ncid, scmlat, scmlon, closelat, closelon,&
@@ -623,6 +628,7 @@ contains
do k=1,2 !loop over months and read vegetated data
+ call t_startf('rmv ncd_io')
call ncd_io(ncid=ncid, varname='MONTHLY_LAI', flag='read', data=mlai, dim1name=grlnd, &
nt=months(k), readvar=readvar)
if (.not. readvar) call endrun(msg=' ERROR: MONTHLY_LAI NOT on fveg file'//errMsg(sourcefile, __LINE__))
@@ -638,11 +644,13 @@ contains
call ncd_io(ncid=ncid, varname='MONTHLY_HEIGHT_BOT', flag='read', data=mhgtb, dim1name=grlnd, &
nt=months(k), readvar=readvar)
if (.not. readvar) call endrun(msg=' ERROR: MONTHLY_HEIGHT_TOP NOT on fveg file'//errMsg(sourcefile, __LINE__))
+ call t_stopf('rmv ncd_io')
! Only vegetated patches have nonzero values
! Assign lai/sai/hgtt/hgtb to the top [maxsoil_patches] patches
! as determined in subroutine surfrd
+ call t_startf('rmv loop')
do p = bounds%begp,bounds%endp
g =patch%gridcell(p)
if (patch%itype(p) /= noveg) then ! vegetated pft
@@ -661,6 +669,7 @@ contains
mhvb2t(p,k) = 0._r8
end if
end do ! end of loop over patches
+ call t_stopf('rmv loop')
end do ! end of loop over months
```
I wasn't able to reproduce quite the extreme results he found: for me, interpMonthlyVeg took roughly 10% of the total lnd run time. I wonder if cheyenne was having glade issues at the time Mike ran? But this is still worth investigating to see if we can reduce this time.
Through my extra timers, I found that nearly all of the time is being spent in the ncd_io calls. I was suspicious of the loop that sets values, because this is written in an inefficient way (there shouldn't be a need for the double nesting of the loop), but the time spent in this loop is negligible compared with the i/o time.
Two initial ideas for getting some speed-up here, which we could address through experimentation and/or consultation with Jim Edwards:
- Would it be more efficient if we arranged the dimensions of these variables as `time, lsmlat, lsmlon, lsmpft` rather than `time, lsmpft, lsmlat, lsmlon`?
- Can we tweak some pio settings to improve the performance of these reads?
Contributor guide
Assessment
This issue has not been assessed yet.