MetOffice / MetOffice/lfric_apps
Optimise LFRic<->UM array copy loop order
- Dominant language
- Fortran
- Stars
- 31
- Forks
- 118
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
### Are there any linked Issues or Pull Requests?
https://github.com/stfc/PSyclone/issues/3609
### Brief description
We investigated the performance of the data copies between LFRic layout (`lfric_array(map(i)+k)`, layers contiguous per column) and UM layout (`um_array(i,1,k)`, cells contiguous per layer) that occur throughout the physics interface. These copies are a small but frequent cost, so we wanted to check whether loop order and OpenMP scheduling choices were leaving performance on the table.
### Further details of the issue
### Microbenchmarking
We built a standalone microbenchmark (MPI + OpenMP, configurable local volume/nlayers, with array cycling to control L3 vs main-memory residency) to test this in isolation from the rest of the model:
- [`lfric_um_layout_cpy`](https://github.com/EPCCed/GungHoKernels2025/tree/main/kernels/lfric_um_layout_cpy): single-array copy, various loop orderings
- [`lfric_um_mult_arr_cpy`](https://github.com/EPCCed/GungHoKernels2025/tree/main/kernels/lfric_um_mult_arr_cpy): multiple arrays copied in the same loop nest
Six loop/parallelisation variants were tested (v0–v5); full details, methodology, and results plots are attached in the PDF ([LFRic2UMresults.pdf](https://github.com/user-attachments/files/32338649/LFRic2UMresults.pdf)). Headline finding: **v4** — outer loop serial over `k`, inner loop over `i` parallelised with `nowait` — gives the best performance across scenarios, avoiding both an L1 thrashing issue (when `k` is innermost with power-of-two cell counts) and a false-sharing issue (when `k` is the parallel loop and writes hit an LFRic array).
> The Cray compiler loop-interchanges v4 back into v1 by default, undoing the benefit. This needed to be suppressed with `-h nointerchange` (or a `!DIR$ NOINTERCHANGE` directive).
### Exploration with application run
As an exploratory test, we applied the v4 pattern to the copy loops in `mphys_kernel_mod.F90`. The snippet below is an illustrative example of the change — the actual default/v4 versions of `mphys_kernel_mod.F90` are attached to this issue([mphys_kernel_mod_test_versions.zip](https://github.com/user-attachments/files/32339825/mphys_kernel_mod_test_versions.zip)).
**Before (illustrative):**
```fortran
!$omp parallel do default(shared) private(i,k) schedule(static)
do k = 1, nlayers
do i = 1, seg_len
um_array(i, 1, k) = real(lfric_array(map(1, i) + k), r_um)
end do
end do
!$omp end parallel do
```
or
```fortran
!$omp parallel do default(shared) private(i,k) schedule(static)
do i = 1, seg_len
do k = 1, nlayers
um_array(i, 1, k) = real(lfric_array(map(1, i) + k), r_um)
end do
end do
!$omp end parallel do
```
The loop ordering is not standardised.
**After (illustrative):**
```fortran
!DIR$ NOINTERCHANGE
!$omp parallel default(shared) private(i,k)
do k = 1, nlayers
!$omp do schedule(static)
!DIR$ UNROLL
do i = 1, seg_len
um_array(i, 1, k) = real(lfric_array(map(1, i) + k), r_um)
end do
!$omp end do nowait
end do
!$omp end parallel
```
Both versions of the file were used to run the `lfric_atm` application with the C224 mesh on Cirrus. The v4 pattern gives some improvement in performance.
### Results
The figure below shows per-timestep timings (rank 0) for the two copy directions, `LFRic → UM` and `UM → LFRic`, comparing the `default` (v1-style, as interchanged by the compiler) and `v4` configurations, for a C224 mesh with 32×32 seglength:
The same data expressed as achieved bandwidth tells the same story from the other side:
Overall, the application-level run follows the microbenchmark finding: the `v4` loop/parallelisation pattern (serial outer loop over `k`, parallel inner loop over `i` with `nowait`, and `NOINTERCHANGE` to stop the compiler undoing it) gives a meaningful reduction in the cost of the LFRic<->UM layout copies in `mphys_kernel_mod.F90`, with the `UM -> LFRic` direction benefiting the most.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with mphys_kernel_mod.F90 and compare its LFRic-to-UM and UM-to-LFRic copy loops with the attached test versions. Review the linked microbenchmark results and run the relevant lfric_atm C224 case on Cirrus to verify that the chosen loop pattern improves copy timings without changing results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- hpc, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100