MetOffice / MetOffice/lfric_apps
Rare generation of NaN (probably) in Comorph
- Dominant language
- Fortran
- Stars
- 31
- Forks
- 118
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
This is about a bug that was tripped during the development of #180, but I don't believe is related to the changes in that PR (it was just unlucky at triggering a rare error).
At the `844de7a68bb2425e91c6f66126ab0752c5453f7f` revision of that branch, the `run_lfric_atm_scm_comorph_dev_toga-BiP2x2-50000x50000_ex1a_cce_fast-debug-32bit` fails, without an error message. Only this test fails, after 1081 timesteps, and this test will succeed on different platforms or with different compiler settings.
After *a lot* of digging, I found that the failure is a segmentation fault in the `qsat_mix` routine called by `mphys_turb_gen_mixed_phase.F90`. However this failure stems from the prognostic fields containing `NaN`s. I have tried to trace the origin of these `NaN`s, and believe that they coming from a routine in comorph: the first `NaN` I can find is in the `updraft_res_source(1,1,45)%res_super` array, in `comorph_main.F90` (45 is the model level of the failure). I have struggled to work further back through the comorph code, so might be wrong.
The failure, with my debugging statements, can be reproduced by running:
```
cylc vip -z group=lfric_atm_scm_comorph_dev_toga-BiP2x2-50000x50000_ex1a_cce_fast-debug-32bit -n debug_high_order_orog ./rose-stem
```
with this branch: https://github.com/tommbendall/lfric_apps/tree/debug_high_order_orog
Things that make this bug particularly challenging to find:
- the lack of traceback from the failure (I obtained that by including the ATP debugging)
- the output log lags behind the model run, and so the last lines appearing in the log can be many timesteps before the actual failure (I have navigated this by logging a lot of extra things to slow the model down)
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with the provided cylc vip command on the debug_high_order_orog branch. Start in comorph_main.F90 by tracing updraft_res_source(1,1,45)%res_super, then follow the path into qsat_mix in mphys_turb_gen_mixed_phase.F90 and inspect when the first NaN appears. Done means the NaN source and triggering conditions are identified and the reproducer no longer fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100