Boundary Layer Umbrella

### Brief description

Create an overarching umbrella issue to capture progress in boundary layer.

We have explicit and implicit files, which seem to have differing optimisation needs:
**bl.explicit:**

- bl_exp_kernel_mod.F90: call bdy_expl2
- bdy_expl2.F90: call btq_int
- bl_exp_kernel_mod.F90: call buoy_tq
- bdy_expl2.F90: call ex_coef
- bdy_expl2.F90: call ex_flux_tq
- bl_exp_du_kernel_mod.F90: call ex_flux_uv
- bdy_expl2.F90: call fm_drag
- bdy_expl2.F90: call kmkhz_9c
- bdy_expl2.F90:call kmkh
- kmkhz_9c.F90: call calc_dqw_inv_wtrac
- kmkhz_9c.F90:call excf_nl_9c
- excf_nl_9c.F90: call excfnl_cci
- excf_nl_9c.F90: call excfnl_compin
- bl_exp_kernel_mod.F90: call tr_mix
- lfric_ukca_callback_mod.f90:call tr_mix
- tr_mix.F90:call imp_mix

** bl.implicit:**

- bl_imp_kernel_mod.F90: call bdy_impl3
- bl_imp2_kernel_mod.F90: call bdy_impl4
- bl_imp2_kernel_mod.F90: call bl_lsp
- Kernel also calls to a number of PC2 schemes

--------------------------------------------------------------------------------------------------------------------------------------------

## PSyclone Issues
Due to some issues with PSyclone, that are not to be rectified in the short term, I have manually adjusted some of the forked files to remove the j loop and improve blocking loop behavior.

--------------------------------------------------------------------------------------------------------------------------------------------
### Hand coded Wave 1:
[Remove j loop bdy_lyr_expl2 #215](https://github.com/MetOffice/lfric_apps/issues/215) - SR
[Remove j loop ex_coef #216](https://github.com/MetOffice/lfric_apps/issues/216) - DONE
[Remove j loop kmkhz_9c #217](https://github.com/MetOffice/lfric_apps/issues/217) - SR
[Remove j loop in excf_nl_9c #219](https://github.com/MetOffice/lfric_apps/issues/219) - DONE

--------------------------------------------------------------------------------------------------------------------------------------------
### Findings For hand coded Wave 2:

Sub dividing this into this wave, and wave 3.
The boundary layer implicit files are struggling with the direct source changes, but the implicit alg/kernel script changes are of good benefit, along with the segment size for explicit.

### Graphs including exploration in wave 3 and changes from wave 1:
Vernier, before and after, explicit routines:

Vernier, before and after, explicit Algorithm top level and Kernel OMP addition:

Vernier, before and after, implicit routines:

Vernier, before and after, implicit Algorithm top level and Kernel(s) OMP addition, and removal of alg script to add OMP to du:

Vernier, before and after, routines which show adverse performance:

CrayPat of same routines which show different picture:

Imbalance stats for fm_drag:

CrayPat OMP overhead before and after:

bdy_imp3, bdy_imp4, ex_flux_uv, imp_mix and tr_mix are also not currently scaling well in the above performance data.
pc2_bl_forced is also called by the kernel, but is not a substantial cost.

Done:
Adjust bdy_lyr local script (required for ex_flux_uv, tr_mix and imp_mix findings) - #255
Turn on the blocking by default - https://github.com/MetOffice/lfric_apps/issues/327 - DONE
Turn on global script for 2x implicit and explicit kernels - https://github.com/MetOffice/lfric_apps/issues/361 - CR
Turn on OMP / Convert du kernels - https://github.com/MetOffice/lfric_apps/issues/362 - DONE
bdy_imp3 - https://github.com/MetOffice/lfric_apps/issues/314 - DONE
bdy_imp4 - https://github.com/MetOffice/lfric_apps/issues/332 - DONE

--------------------------------------------------------------------------------------------------------------------------------------------
### Findings For hand coded Wave 3:

Findings indicated that blocking for implicit using what is working for explicit is not ideal. This was broken out to help reviews, but results are rolled back into the respective routine PR.
This has been explored, and it seems that existing threading behaviors for implicit routines are the best means forward.
It's not totally clear, and could do with another look once everything on, but there will be more expensive OMP sections to look at.

DONE:
Universalise blocking across the boundary layer - https://github.com/MetOffice/lfric_apps/issues/334 - Explored with latest updates, best seems to be on trunk
pc2_bl_forced - https://github.com/MetOffice/lfric_apps/issues/333 - DONE already by Exeter Uni RSE

Follow up work:
Possible re-exploration of segment sizes? Chris advised that when a internal k loop runs backwards, inside blocking is smashes cache reuse, hence why the blocking over a wider range is better here. When it's forward, the normal blocking off a segment which is smaller is likely better.

SR Feedback:
From #377: The looping and flow control is probably not the most efficient. In particular the whether i or k is inner most and which one has the openMP means there are lots of little parallel regions. This can be improved but is not in the scope of this ticket and is a follow up. Also bl_imp2_kernel calls into the PC2 scheme, and it least one of those routines has hand-coded OpenMP around j, which is one. Again, out of scope for this ticket.

--------------------------------------------------------------------------------------------------------------------------------------------
### Extras to explore further by hand

Given the blocking behavior on some larger loops is showing improvement, these 3x schemes could be looked at (amount the ones done for j and dynamic a similar data pattern):
- science/physics_schemes/source/large_scale_cloud/pc2_bl_inhom_ice.F90
- science/physics_schemes/source/boundary_layer/bl_lsp.F90
- science/physics_schemes/source/large_scale_cloud/pc2_delta_hom_turb.F90

--------------------------------------------------------------------------------------------------------------------------------------------

## For PSyclone, Wave 1:
(https://github.com/MetOffice/lfric_apps/issues/50) - DONE
(https://github.com/MetOffice/lfric_apps/issues/49) - DONE

--------------------------------------------------------------------------------------------------------------------------------------------
## For PSyclone, Wave 2:
PSyclone is struggling in some files. Longer term we need to be using PSyclone on all Physics source for Hardware portable source. Whilst it's functionality is being expanded, as edge cases are found, there is also an argument for refactoring code to make this easier for PSyclone.
https://github.com/MetOffice/lfric_apps/issues/560
https://github.com/MetOffice/lfric_apps/issues/571

--------------------------------------------------------------------------------------------------------------------------------------------