CliMA / CliMA/EnsembleKalmanProcesses.jl

SparseInversion not as robust as Inversion

Open
#134 5 comments 0 reactions 0 assignees View on GitHub
robustness
Dominant language
Julia
Stars
125
Forks
24
Avg merge
1d 18h
Merged PRs (30d)
5

Description

I tried using the SparseInversion algorithm instead of Inversion for an integration test in [CEDMF.jl](https://github.com/CliMA/CalibrateEDMF.jl/pull/272), and I run into a `SingularException` in [L111](https://github.com/CliMA/EnsembleKalmanProcesses.jl/blob/main/src/SparseEnsembleKalmanInversion.jl#L111).

The configuration used for the integration test is

```
γ = 1.0
threshold_eki = true
threshold_value = 5e-2
reg = 1e-2
```

In general, we are missing documentation for how to choose `reg` and how to make the convex optimization part of the update robust.

Stack trace:

```
Stacktrace:
--
  | [1] checknonsingular
  | @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/factorization.jl:19 [inlined]
  | [2] checknonsingular
  | @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/factorization.jl:21 [inlined]
  | [3] #lu!#146
  | @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/lu.jl:82 [inlined]
  | [4] #lu#153
  | @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/lu.jl:279 [inlined]
  | [5] lu (repeats 2 times)
  | @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/lu.jl:278 [inlined]
  | [6] \(A::Matrix{Float64}, B::Diagonal{Float64, Vector{Float64}})
  | @ LinearAlgebra /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/generic.jl:1142
  | [7] sparse_eki_update(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64, Int64}}, u::Matrix{Float64}, g::Matrix{Float64}, y::Matrix{Float64}, obs_noise_cov::Matrix{Float64})
  | @ EnsembleKalmanProcesses /central/scratch/esm/slurm-buildkite/calibrateedmf-ci/depot/cpu/packages/EnsembleKalmanProcesses/OjLuD/src/SparseEnsembleKalmanInversion.jl:111
  | [8] (::EnsembleKalmanProcesses.var"#failsafe_update#26")(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64, Int64}}, u::Matrix{Float64}, g::Matrix{Float64}, y::Matrix{Float64}, obs_noise_cov::Matrix{Float64}, failed_ens::Vector{Int64})
  | @ EnsembleKalmanProcesses /central/scratch/esm/slurm-buildkite/calibrateedmf-ci/depot/cpu/packages/EnsembleKalmanProcesses/OjLuD/src/SparseEnsembleKalmanInversion.jl:24
  | [9] update_ensemble!(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64, Int64}}, g::Matrix{Float64}; cov_threshold::Float64, Δt_new::Float64, deterministic_forward_map::Bool, failed_ens::Nothing)
  | @ EnsembleKalmanProcesses /central/scratch/esm/slurm-buildkite/calibrateedmf-ci/depot/cpu/packages/EnsembleKalmanProcesses/OjLuD/src/SparseEnsembleKalmanInversion.jl:191

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/SparseEnsembleKalmanInversion.jl at line 111 and reproduce the SingularException with the configuration and CEDMF integration test described in the issue. Determine how the SparseInversion update should handle this case, and document how to choose reg and make the convex optimization update robust.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.