MetOffice / MetOffice/lfric_apps
Explore OMP issues in GH calipers
- Dominant language
- Fortran
- Stars
- 31
- Forks
- 118
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
### Are there any linked Issues or Pull Requests?
Umbrella: #779
PR: #783
### Brief description
Some GHM calipers (see the comment below) are showing some scaling issues with OMP at higher threads on Genoa.
For the most part, OMP coverage is present across all of the algorithm's generated psy layers loops.
As you can see in the C192 performance data below, 1,2 and 4 threads, whilst starting to get worse, only gets noticeable at 8 or 16 threads.
I think in these instances, we might need to look at whether these need some blocking issues to aid with cache usage, even with Genoa.
There is some coverage missing from one of the algorithms, `ffsl_advective_updates_alg_mod`. Originally this was a choice, however it is believed that this is thread safe, and so we are looking at reactivating it our PR here: https://github.com/MetOffice/lfric_apps/pull/783
| Before/After | Caliper | Min | Mean | SMax | SMin | SMean | - | - | - | - |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| vern_comb_run_3n_16T_run1.txt: | transport.ffsl_vertical | 11.014 | 11.27437 | 11.547 | 10.1374 | 10.37657 | 10.597 | 7440 | 3.23745 | 0.00152 |
| vern_comb_run_3n_1T_run1.txt: | transport.ffsl_vertical | 4.27244 | 4.37215 | 4.60479 | 3.71 | 3.79582 | 3.99816 | 7440 | 1.9717 | 0.00059 |
| vern_comb_run_3n_2T_run1.txt: | transport.ffsl_vertical | 4.67359 | 4.76566 | 4.86713 | 4.06031 | 4.13774 | 4.23039 | 7440 | 2.15581 | 0.00064 |
| vern_comb_run_3n_4T_run1.txt: | transport.ffsl_vertical | 5.37735 | 5.44951 | 5.56826 | 4.84646 | 4.91544 | 5.03611 | 7440 | 2.4174 | 0.00073 |
| vern_comb_run_3n_8T_run1.txt: | transport.ffsl_vertical | 6.18233 | 7.00357 | 7.23576 | 5.67111 | 6.43244 | 6.66235 | 7440 | 2.72927 | 0.00094 |
| vern_comb_run_3n_16T_run1.txt: | pressure_solver.helmholtz_lhs | 4.33701 | 5.60761 | 7.41249 | 2.08851 | 2.24446 | 2.48091 | 29325 | 0.70026 | 0.00019 |
| vern_comb_run_3n_1T_run1.txt: | pressure_solver.helmholtz_lhs | 2.82598 | 4.87902 | 6.6355 | 1.10607 | 1.28037 | 1.43164 | 29325 | 0.66507 | 0.00017 |
| vern_comb_run_3n_2T_run1.txt: | pressure_solver.helmholtz_lhs | 3.1133 | 4.04234 | 4.79014 | 1.25839 | 1.4118 | 1.54814 | 29340 | 0.73556 | 0.00014 |
| vern_comb_run_3n_4T_run1.txt: | pressure_solver.helmholtz_lhs | 2.92865 | 4.04367 | 4.84265 | 1.58733 | 1.67359 | 1.81359 | 29325 | 0.82307 | 0.00014 |
| vern_comb_run_3n_8T_run1.txt: | pressure_solver.helmholtz_lhs | 3.25966 | 3.90147 | 6.57733 | 1.2989 | 1.85503 | 1.99559 | 29325 | 0.78709 | 0.00013 |
| vern_comb_run_3n_16T_run1.txt: | mappings.set_wind | 14.2831 | 16.55898 | 20.5787 | 1.03635 | 1.2171 | 1.40771 | 481 | 0.37973 | 0.03443 |
| vern_comb_run_3n_1T_run1.txt: | mappings.set_wind | 2.74615 | 5.6197 | 10.4265 | 0.29536 | 0.4471 | 0.70586 | 481 | 0.23224 | 0.01168 |
| vern_comb_run_3n_2T_run1.txt: | mappings.set_wind | 3.33395 | 5.54409 | 9.0347 | 0.37971 | 0.53875 | 0.73158 | 481 | 0.2807 | 0.01153 |
| vern_comb_run_3n_4T_run1.txt: | mappings.set_wind | 4.12135 | 6.18357 | 8.74031 | 0.54091 | 0.69322 | 0.86513 | 481 | 0.34093 | 0.01286 |
| vern_comb_run_3n_8T_run1.txt: | mappings.set_wind | 10.9954 | 13.20816 | 19.8334 | 0.68455 | 0.91767 | 1.0383 | 481 | 0.38937 | 0.02746 |
### Further details of the issue.
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with umbrella issue #779 and PR #783, then review the Genoa Caliper results for transport.ffsl_vertical, pressure_solver.helmholtz_lhs, and mappings.set_wind. Check the OMP coverage for ffsl_advective_updates_alg_mod and determine which higher-thread scaling issues need investigation. Done means documenting the affected calipers and a justified direction for improving their scaling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- hpc, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100