NCAR / NCAR/DART

Performance of EAKF when the sd is 0

Open
#932 5 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

performance pmp
Dominant language
Fortran
Stars
263
Forks
182
Avg merge
11d 12h
Merged PRs (30d)
7

Description

When using the QCEFF with BNRH there is an early exit from the sequential_obs loop when the probit transform fails:

https://github.com/NCAR/DART/blob/ba604a564e5950d74b78c6b057da13ec5ebc01ee/assimilation_code/modules/assimilation/assim_tools_mod.f90#L705-L715

to_probit_bounded_normal_rh fails it sd <= 0.0_r8 (so -99 failed, or 0). For observations such as radar_clearair_refractivity, there are lots of forward operators where the result is 0 across the ensemble. When this happens the fwd ops fail to_probit_bounded_normal_rh

Image

Bailing out of sequential_obs at this stage saves a lot of unnecessary get_close calls and therefore a lot of runtime. I believe the get_close calls are unnecessary because if there is no spread,
https://github.com/NCAR/DART/blob/ba604a564e5950d74b78c6b057da13ec5ebc01ee/assimilation_code/modules/assimilation/assim_tools_mod.f90#L1497-L1501
the assimilation increments are zero
https://github.com/NCAR/DART/blob/ba604a564e5950d74b78c6b057da13ec5ebc01ee/assimilation_code/modules/assimilation/assim_tools_mod.f90#L1548

so no need to do any get_close to find state to not update.

Here is a profiling pic where about half the obs are 0 spread. Eakf is top, same shape, but you can see sequential obs do (the green bit) is stretched.
Image

Note this effect is amplified by the get_close_initialization #910 cost, but even if you remove that there is still a considerable performance penalty for eakf vs bnrh when there are bunch of fwd ops with sd = 0.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.