Slow get_expected_radiance() with MPAS (redundant calls to find cellID?)

Hi,
We found that get_expected_radiance() in obs_def_rttov_mod.f90 is slow with MPAS but not WRF.
In my tests, the model_interpolate() block L3603 - L3920 takes ~80% of the walltime per observation (0.56-0.94s) while the call to rttov_direct() (the RTTOV function) is < 0.001 sec per obs.
https://github.com/NCAR/DART/blob/146c853f005568cdc41272e04f5dd85aeb18de74/observations/forward_operators/obs_def_rttov13_mod.f90#L3603

Can it be that the search is performed again for every variable and vertical level that is requested?
At least in WRF and MPAS, the cell index is the same for all vertical layers and variables, so it only needs to be computed once for e.g. 20 variables x 100 layers.
https://github.com/NCAR/DART/blob/146c853f005568cdc41272e04f5dd85aeb18de74/models/mpas_atm/model_mod.f90#L6388

Could you take a look at this? Thanks a lot!
Lukas

No issue in WRF

For comparison, I timed perfect_model_obs() with MPAS on derecho. 200 obs took 3 minutes and the interpolation block (= bottleneck) took 0.81 (+/- 0.005) seconds per observation (each obs at the same location for simplicity). On the Vienna cluster, PMO with WRF and 200 obs took 11s. Single process in both cases, but WRF and MPAS were different domains.

Error Message

None.

Which model(s) are you working with?

MPAS_atm

Version of DART

I compared (freshly compiled) v10.1.0 and v11.17.0.
For 200 observations, timings are 3:55 and 3:25 respectively (min:sec) on derecho.

Have you modified the DART code?

No

Start in observations/forward_operators/obs_def_rttov13_mod.f90 around lines 3603–3920 and inspect the model_interpolate() block. Compare its behavior with models/mpas_atm/model_mod.f90 around line 6388, focusing on whether cell-index searches repeat across variables and vertical levels. Use the reported perfect_model_obs() timings for MPAS and WRF as the baseline; done means the source of the bottleneck is identified and the performance impact is validated.