NCAR / NCAR/MPAS-Workflow

If VarBC is activated, EDA scenario is failing at the 2nd cycle of DA task

Open
#416 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
25
Forks
19
Avg merge
1d 49m
Merged PRs (30d)
2

Description

Describe the bug
VarBC requires the prior information of coefficient and its covariance. At the first cycle, the specified static files are used as a prior information. Later, the previous analyzed VarBC coefficient and its covariance files are used as a prior information at current DA cycle.

For EDA scenario, the path for prior information is not populated properly and this leads to a failure of DA task at the 2nd cycle.

To Reproduce
This can be reproduced by running test/testinput/eda_OIE120km_WarmStart.yaml with adding biasCorrection: True under variational key.
The example of error message from variational runs at the 2nd cycle is as below:

Exception:      Reason: H5Fopen failed
        compat: HDF5_Version_Range: [V110, Latest (1.14.3)]
        filename:       /glade/derecho/scratch/bjung/pandac/bjung_eda_OIE120km_WarmStart_TEST_PR409_VarBC/CyclingDA/2018041500/dbOut/satbias_cov_amsua_aqua.h5
        mode:   ioda::Engines::BackendOpenModes::Read_Only
        source_column:  0
        source_filename:        /glade/work/jwittig/repos1/mpas-bundle-cron-src/mpas-bundle/ioda/src/engines/ioda/src/ioda/Engines/HH/HH.cpp
        source_function:        ioda::Group ioda::Engines::HH::openFile(const std::string&, ioda::Engines::BackendOpenModes, HDF5_Version_Range)
        source_line:    185
        stacktrace:
 0# util::stacktrace_current[abi:cxx11]() in /glade/derecho/scratch/jwittig/repos-s/mpas-bundle-cron/build-gnu-1p_02_07_26/bin/../lib64/liboops.so
 1# ioda::Exception::add_call_stack() in /glade/derecho/scratch/jwittig/repos-s/mpas-bundle-cron/build-gnu-1p_02_07_26/bin/../lib64/libioda_engines.so
 2# ioda::Exception::Exception(char const*, ioda::detail::compat::source_location::source_location const&, ioda::Options const&) in /glade/derecho/scratch/jwittig/repos-s/mpas-bundle-cron/build-gnu-1p_02_07_26/bin/../lib64/libioda_engines.so
 3# 0x0000148B55BCC283 in /glade/derecho/scratch/jwittig/repos-s/mpas-bundle-cron/build-gnu-1p_02_07_26/bin/../lib64/libioda_engines.so
 4# ioda::Engines::constructBackend(ioda::Engines::BackendNames, ioda::Engines::BackendCreationParameters&) in /glade/derecho/scratch/jwittig/repos-s/mpas-bundle-cron/build-gnu-1p_02_07_26/bin/../lib64/libioda_engines.so
 5# ufo::ObsBiasCovariance::read(eckit::Configuration const&) in /glade/derecho/scratch/jwittig/repos-s/mpas-bundle-cron/build-gnu-1p_02_07_26/bin/../lib64/libufo.so
 6# ufo::ObsBiasCovariance::ObsBiasCovariance(ioda::ObsSpace&, eckit::Configuration const&) in /glade/derecho/scratch/jwittig/repos-s/mpas-bundle-cron/build-gnu-1p_02_07_26/bin/../lib64/libufo.so
 7# 0x00000000005B60B1 in ./mpasjedi_variational.x
 8# 0x00000000005D589C in ./mpasjedi_variational.x
 9# 0x00000000005D5A58 in ./mpasjedi_variational.x
10# 0x0000000000665C54 in ./mpasjedi_variational.x
11# 0x0000000000675F95 in ./mpasjedi_variational.x
12# 0x0000000000676495 in ./mpasjedi_variational.x
13# 0x00000000005769AE in ./mpasjedi_variational.x
14# 0x0000000000676576 in ./mpasjedi_variational.x
15# oops::Run::execute(oops::Application const&, eckit::mpi::Comm const&) in /glade/derecho/scratch/jwittig/repos-s/mpas-bundle-cron/build-gnu-1p_02_07_26/bin/../lib64/liboops.so
16# 0x000000000047125B in ./mpasjedi_variational.x
17# 0x0000148B4F556E6C in /lib64/libc.so.6
18# __libc_start_main in /lib64/libc.so.6
19# 0x00000000004712A1 in ./mpasjedi_variational.x

Expected behavior
EDA can be run with VarBC. But also wondering if it is desirable.

potential fix
We need to consider multiple members (if exist) in https://github.com/NCAR/MPAS-Workflow/blob/develop/bin/PrepJEDI.csh#L278 https://github.com/NCAR/MPAS-Workflow/blob/develop/bin/PrepJEDI.csh#L347-L348 , similar to https://github.com/NCAR/MPAS-Workflow/blob/develop/bin/PrepJEDI.csh#L596-L601 .

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running test/testinput/eda_OIE120km_WarmStart.yaml with biasCorrection: True under variational, then inspect bin/PrepJEDI.csh around lines 347-348. Compare its handling of multiple members with the logic around lines 596-601. Done means the EDA scenario reaches its second cycle without the missing VarBC covariance path, subject to confirming that running EDA with VarBC is desired.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.