MetOffice / MetOffice/lfric_apps
using lfric2lfric for C896 to UKV
- Dominant language
- Fortran
- Stars
- 31
- Forks
- 118
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
### Are there any linked Issues or Pull Requests?
- MetOffice/lfric_apps#324
- MetOffice/lfric_apps#275
- MetOffice/lfric_apps#238
- MetOffice/lfric_apps#201
### Brief description
A number of tickets related to lfric2lfric are in progress. A branch that includes the above PRs has been created to enable testing: https://github.com/MetOffice/lfric_apps/compare/main...ukmo-juan-castillo:lfric_apps:lfric2lfric_horizontal_merged
The purpose of this issue is to document the changes that are required so that the branch can regrid from C896 (global) to UKV LAM (regional).
### Further details of the issue.
Local changes to this branch are at /home/users/christine.johnson/git/ukmo-juan-castillo/lfric_apps
Changes required are:
* The gen_weights tasks doesn't work correctly with a multigrid mesh file, so single mesh files are specified in
`rose-stem/templates/runtime/generate_runtime_gen_weights.cylc`. A new C896 mesh file config is created at `rose-stem/app/mesh/opt/rose-app-C896.conf`.
* In `applications/lfric2lfric/source/initialisation/lfric2lfric_init_mesh.f90` the mesh reading tasks are reordered and clear global mesh calls are included, to prevent out of memory problems. The old code reads the target mesh and then the source mesh and then creates the local meshes. The new code reads the source mesh, creates the local mesh, clears the global mesh. Then reads the target mesh, creates the local mesh, clears the global mesh.
* In `rose-stem/app/generate_weights/opt/rose-app-lfric2lfric.conf ` this now has `INT_METHOD=conserve2nd` rather than `conserve` to enable a higher order horizontal interpolation method.
* In `rose-stem/templates/default_directives.cylc` the `gen_weights_tasks` is increased from `8` to `16`.
* To prevent the source file being copied to the working directory (which is painfully slow for a large file!), `mode=symlink` is added to `[file:...]` in `rose-stem/app/lfric2lfric/opt/rose-app-clim_gal9_ral_C896_ukv.conf`. This creates a soft link from the working directory to the source file directory.
* In `rose-stem/site/common/lfric2lfric/tasks_lfric2lfric.cylc` task `oasis_C896_ukv_lam` uses mpi_parts 768, and memory 250.
* In `rose-stem/app/lfric2lfric/opt/rose-app-multi-cpu_C896_ukv.conf` the cubedsphere partitioner uses `panel_xproc=8` and
`panel_yproc=16`.
* `rose-stem/app/lfric2lfric/file/iodef.xml` has been adjusted to include fields defined with `grid_ref=full_level_face_grid` and `half_level_face_grid` in the checkpoint_read, rather than `domain_ref=checkpoint_W3 and checkpoint_Wtheta`. This is because the C896 source file is an LFRic diagnostics file, rather than a checkpoint file.
* The required meshes are only created by the suite if they are listed under `resolution` in `rose-stem/site/common/lfric2lfric/tasks_lfric2lfric.cylc` rather than `dst_mesh` and `src_mesh`.
In summary:
| Source | Target | mpi | memory (GB) | panel_xproc| panel_yproc | number fields regridded | time taken |
| ------- | ----------------- | ----- | --------- | ------------ | ------------- | --------------------------- | ---------- |
| C896 diagnostics file | UKV (1024x960) | 768 | 250 | 8 | 16 | 9 (theta, rho, p, u_in_w3, v_in_w3, mr, mcl, mv, mcf) | 10 mins |
| C896 diagnostics file | UKV (1024x960) | 384 | 250 | 8 | 8 | 1 (theta) | 3 mins |
| C224 checkpoint file | 648x648 | 36 | 250 | 2 | 3 | 3 (theta, rho, zh) | |
| C224 checkpoint file | 672x672 | 36 | 250 | 2 | 3 | 3 (theta, rho, zh) | |
| C224 checkpoint file | UKV (1024x960) | 48 | 250 | 2 | 4 | 3 (theta, rho, zh) | |
| C224 checkpoint file | UKV (1024x960) | 384 | 250 | 8 | 8 | 33 theta, rho, m_v, m_cl, m_r, m_ci, area_fraction, bulk_fraction, liquid_fraction, frozen_fraction, tile_snow_mass, n_snow_layers, snow_depth, tile_snow_r_grain, snowpack_density, snow_under_canopy, canopy_water, tile_temperature, snow_layer_thickness, snow_layer_ice_mass, snow_layer_liq_mass, snow_layer_temp, snow_layer_rgrain, soil_temperature, soil_moisture, zh, z0msea, dd_ff_cb, snow_soot, soil_sat_frac, water_table, wetness_under_soil, soil_thermal_cond | 12 mins |
Remaining problems:
* The code is successful in writing data to a UKV checkpoint file. However, there seems to be a problem with the coordinates when trying to plot the data. I'm using iris.load to load in the file, and this works fine for the global file. But for the lam file I can't get it to read the coordinates. It says "*Assuming* 'topology_dimension'=0, from the attached connectivities of the mesh variable checkpoint_Wtheta. However, checkpoint_Wtheta:topology_dimension = 2 -- ignoring this as it is inconsistent."
So I'm using netcdf4 to read the coordinates which seems to be fine for plotting.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.