MetOffice / MetOffice/lfric_apps

Rare generation of NaN (probably) in Comorph

Open
#550 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Fortran
Stars
31
Forks
118
Avg merge
2d 4h
Merged PRs (30d)
8

Description

This is about a bug that was tripped during the development of #180, but I don't believe is related to the changes in that PR (it was just unlucky at triggering a rare error).

At the `844de7a68bb2425e91c6f66126ab0752c5453f7f` revision of that branch, the `run_lfric_atm_scm_comorph_dev_toga-BiP2x2-50000x50000_ex1a_cce_fast-debug-32bit` fails, without an error message. Only this test fails, after 1081 timesteps, and this test will succeed on different platforms or with different compiler settings.

After *a lot* of digging, I found that the failure is a segmentation fault in the `qsat_mix` routine called by `mphys_turb_gen_mixed_phase.F90`. However this failure stems from the prognostic fields containing `NaN`s. I have tried to trace the origin of these `NaN`s, and believe that they coming from a routine in comorph: the first `NaN` I can find is in the `updraft_res_source(1,1,45)%res_super` array, in `comorph_main.F90` (45 is the model level of the failure). I have struggled to work further back through the comorph code, so might be wrong.

The failure, with my debugging statements, can be reproduced by running:
```
cylc vip -z group=lfric_atm_scm_comorph_dev_toga-BiP2x2-50000x50000_ex1a_cce_fast-debug-32bit -n debug_high_order_orog ./rose-stem
```
with this branch: https://github.com/tommbendall/lfric_apps/tree/debug_high_order_orog

Things that make this bug particularly challenging to find:
- the lack of traceback from the failure (I obtained that by including the ATP debugging)
- the output log lags behind the model run, and so the last lines appearing in the log can be many timesteps before the actual failure (I have navigated this by logging a lot of extra things to slow the model down)

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with the provided cylc vip command on the debug_high_order_orog branch. Start in comorph_main.F90 by tracing updraft_res_source(1,1,45)%res_super, then follow the path into qsat_mix in mphys_turb_gen_mixed_phase.F90 and inspect when the first NaN appears. Done means the NaN source and triggering conditions are identified and the reproducer no longer fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.