MetOffice / MetOffice/lfric_apps

Explore OMP issues in GH calipers

Open
#784 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Fortran
Stars
31
Forks
118
Avg merge
2d 4h
Merged PRs (30d)
8

Description

### Are there any linked Issues or Pull Requests?

Umbrella: #779
PR: #783

### Brief description

Some GHM calipers (see the comment below) are showing some scaling issues with OMP at higher threads on Genoa.

For the most part, OMP coverage is present across all of the algorithm's generated psy layers loops.
As you can see in the C192 performance data below, 1,2 and 4 threads, whilst starting to get worse, only gets noticeable at 8 or 16 threads.
I think in these instances, we might need to look at whether these need some blocking issues to aid with cache usage, even with Genoa.

There is some coverage missing from one of the algorithms, `ffsl_advective_updates_alg_mod`. Originally this was a choice, however it is believed that this is thread safe, and so we are looking at reactivating it our PR here: https://github.com/MetOffice/lfric_apps/pull/783

| Before/After | Caliper | Min | Mean | SMax | SMin | SMean | - | - | - | - |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| vern_comb_run_3n_16T_run1.txt: | transport.ffsl_vertical | 11.014 | 11.27437 | 11.547 | 10.1374 | 10.37657 | 10.597 | 7440 | 3.23745 | 0.00152 |
| vern_comb_run_3n_1T_run1.txt: | transport.ffsl_vertical | 4.27244 | 4.37215 | 4.60479 | 3.71 | 3.79582 | 3.99816 | 7440 | 1.9717 | 0.00059 |
| vern_comb_run_3n_2T_run1.txt: | transport.ffsl_vertical | 4.67359 | 4.76566 | 4.86713 | 4.06031 | 4.13774 | 4.23039 | 7440 | 2.15581 | 0.00064 |
| vern_comb_run_3n_4T_run1.txt: | transport.ffsl_vertical | 5.37735 | 5.44951 | 5.56826 | 4.84646 | 4.91544 | 5.03611 | 7440 | 2.4174 | 0.00073 |
| vern_comb_run_3n_8T_run1.txt: | transport.ffsl_vertical | 6.18233 | 7.00357 | 7.23576 | 5.67111 | 6.43244 | 6.66235 | 7440 | 2.72927 | 0.00094 |
| vern_comb_run_3n_16T_run1.txt: | pressure_solver.helmholtz_lhs | 4.33701 | 5.60761 | 7.41249 | 2.08851 | 2.24446 | 2.48091 | 29325 | 0.70026 | 0.00019 |
| vern_comb_run_3n_1T_run1.txt: | pressure_solver.helmholtz_lhs | 2.82598 | 4.87902 | 6.6355 | 1.10607 | 1.28037 | 1.43164 | 29325 | 0.66507 | 0.00017 |
| vern_comb_run_3n_2T_run1.txt: | pressure_solver.helmholtz_lhs | 3.1133 | 4.04234 | 4.79014 | 1.25839 | 1.4118 | 1.54814 | 29340 | 0.73556 | 0.00014 |
| vern_comb_run_3n_4T_run1.txt: | pressure_solver.helmholtz_lhs | 2.92865 | 4.04367 | 4.84265 | 1.58733 | 1.67359 | 1.81359 | 29325 | 0.82307 | 0.00014 |
| vern_comb_run_3n_8T_run1.txt: | pressure_solver.helmholtz_lhs | 3.25966 | 3.90147 | 6.57733 | 1.2989 | 1.85503 | 1.99559 | 29325 | 0.78709 | 0.00013 |
| vern_comb_run_3n_16T_run1.txt: | mappings.set_wind | 14.2831 | 16.55898 | 20.5787 | 1.03635 | 1.2171 | 1.40771 | 481 | 0.37973 | 0.03443 |
| vern_comb_run_3n_1T_run1.txt: | mappings.set_wind | 2.74615 | 5.6197 | 10.4265 | 0.29536 | 0.4471 | 0.70586 | 481 | 0.23224 | 0.01168 |
| vern_comb_run_3n_2T_run1.txt: | mappings.set_wind | 3.33395 | 5.54409 | 9.0347 | 0.37971 | 0.53875 | 0.73158 | 481 | 0.2807 | 0.01153 |
| vern_comb_run_3n_4T_run1.txt: | mappings.set_wind | 4.12135 | 6.18357 | 8.74031 | 0.54091 | 0.69322 | 0.86513 | 481 | 0.34093 | 0.01286 |
| vern_comb_run_3n_8T_run1.txt: | mappings.set_wind | 10.9954 | 13.20816 | 19.8334 | 0.68455 | 0.91767 | 1.0383 | 481 | 0.38937 | 0.02746 |

### Further details of the issue.

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with umbrella issue #779 and PR #783, then review the Genoa Caliper results for transport.ffsl_vertical, pressure_solver.helmholtz_lhs, and mappings.set_wind. Check the OMP coverage for ffsl_advective_updates_alg_mod and determine which higher-thread scaling issues need investigation. Done means documenting the affected calipers and a justified direction for improving their scaling.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
hpc, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.