MetOffice / MetOffice/lfric_apps

Optimise LFRic<->UM array copy loop order

Open
#793 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Fortran
Stars
31
Forks
118
Avg merge
2d 4h
Merged PRs (30d)
8

Description

### Are there any linked Issues or Pull Requests?

https://github.com/stfc/PSyclone/issues/3609

### Brief description

We investigated the performance of the data copies between LFRic layout (`lfric_array(map(i)+k)`, layers contiguous per column) and UM layout (`um_array(i,1,k)`, cells contiguous per layer) that occur throughout the physics interface. These copies are a small but frequent cost, so we wanted to check whether loop order and OpenMP scheduling choices were leaving performance on the table.

### Further details of the issue

### Microbenchmarking

We built a standalone microbenchmark (MPI + OpenMP, configurable local volume/nlayers, with array cycling to control L3 vs main-memory residency) to test this in isolation from the rest of the model:

- [`lfric_um_layout_cpy`](https://github.com/EPCCed/GungHoKernels2025/tree/main/kernels/lfric_um_layout_cpy): single-array copy, various loop orderings
- [`lfric_um_mult_arr_cpy`](https://github.com/EPCCed/GungHoKernels2025/tree/main/kernels/lfric_um_mult_arr_cpy): multiple arrays copied in the same loop nest

Six loop/parallelisation variants were tested (v0–v5); full details, methodology, and results plots are attached in the PDF ([LFRic2UMresults.pdf](https://github.com/user-attachments/files/32338649/LFRic2UMresults.pdf)). Headline finding: **v4** — outer loop serial over `k`, inner loop over `i` parallelised with `nowait` — gives the best performance across scenarios, avoiding both an L1 thrashing issue (when `k` is innermost with power-of-two cell counts) and a false-sharing issue (when `k` is the parallel loop and writes hit an LFRic array).

> The Cray compiler loop-interchanges v4 back into v1 by default, undoing the benefit. This needed to be suppressed with `-h nointerchange` (or a `!DIR$ NOINTERCHANGE` directive).

### Exploration with application run

As an exploratory test, we applied the v4 pattern to the copy loops in `mphys_kernel_mod.F90`. The snippet below is an illustrative example of the change — the actual default/v4 versions of `mphys_kernel_mod.F90` are attached to this issue([mphys_kernel_mod_test_versions.zip](https://github.com/user-attachments/files/32339825/mphys_kernel_mod_test_versions.zip)).

**Before (illustrative):**
```fortran
!$omp parallel do default(shared) private(i,k) schedule(static)
do k = 1, nlayers
do i = 1, seg_len
um_array(i, 1, k) = real(lfric_array(map(1, i) + k), r_um)
end do
end do
!$omp end parallel do
```

or

```fortran
!$omp parallel do default(shared) private(i,k) schedule(static)
do i = 1, seg_len
do k = 1, nlayers
um_array(i, 1, k) = real(lfric_array(map(1, i) + k), r_um)
end do
end do
!$omp end parallel do
```
The loop ordering is not standardised.

**After (illustrative):**
```fortran
!DIR$ NOINTERCHANGE
!$omp parallel default(shared) private(i,k)
do k = 1, nlayers
!$omp do schedule(static)
!DIR$ UNROLL
do i = 1, seg_len
um_array(i, 1, k) = real(lfric_array(map(1, i) + k), r_um)
end do
!$omp end do nowait
end do
!$omp end parallel
```

Both versions of the file were used to run the `lfric_atm` application with the C224 mesh on Cirrus. The v4 pattern gives some improvement in performance.

### Results

The figure below shows per-timestep timings (rank 0) for the two copy directions, `LFRic → UM` and `UM → LFRic`, comparing the `default` (v1-style, as interchanged by the compiler) and `v4` configurations, for a C224 mesh with 32×32 seglength:

Image

The same data expressed as achieved bandwidth tells the same story from the other side:

Image

Overall, the application-level run follows the microbenchmark finding: the `v4` loop/parallelisation pattern (serial outer loop over `k`, parallel inner loop over `i` with `nowait`, and `NOINTERCHANGE` to stop the compiler undoing it) gives a meaningful reduction in the cost of the LFRic<->UM layout copies in `mphys_kernel_mod.F90`, with the `UM -> LFRic` direction benefiting the most.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with mphys_kernel_mod.F90 and compare its LFRic-to-UM and UM-to-LFRic copy loops with the attached test versions. Review the linked microbenchmark results and run the relevant lfric_atm C224 case on Cirrus to verify that the chosen loop pattern improves copy timings without changing results.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
hpc, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.