CliMA / CliMA/EnsembleKalmanProcesses.jl
SparseInversion not as robust as Inversion
- Dominant language
- Julia
- Stars
- 125
- Forks
- 24
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 5
Description
I tried using the SparseInversion algorithm instead of Inversion for an integration test in [CEDMF.jl](https://github.com/CliMA/CalibrateEDMF.jl/pull/272), and I run into a `SingularException` in [L111](https://github.com/CliMA/EnsembleKalmanProcesses.jl/blob/main/src/SparseEnsembleKalmanInversion.jl#L111).
The configuration used for the integration test is
```
γ = 1.0
threshold_eki = true
threshold_value = 5e-2
reg = 1e-2
```
In general, we are missing documentation for how to choose `reg` and how to make the convex optimization part of the update robust.
Stack trace:
```
Stacktrace:
--
| [1] checknonsingular
| @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/factorization.jl:19 [inlined]
| [2] checknonsingular
| @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/factorization.jl:21 [inlined]
| [3] #lu!#146
| @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/lu.jl:82 [inlined]
| [4] #lu#153
| @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/lu.jl:279 [inlined]
| [5] lu (repeats 2 times)
| @ /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/lu.jl:278 [inlined]
| [6] \(A::Matrix{Float64}, B::Diagonal{Float64, Vector{Float64}})
| @ LinearAlgebra /central/software/julia/1.7.0/share/julia/stdlib/v1.7/LinearAlgebra/src/generic.jl:1142
| [7] sparse_eki_update(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64, Int64}}, u::Matrix{Float64}, g::Matrix{Float64}, y::Matrix{Float64}, obs_noise_cov::Matrix{Float64})
| @ EnsembleKalmanProcesses /central/scratch/esm/slurm-buildkite/calibrateedmf-ci/depot/cpu/packages/EnsembleKalmanProcesses/OjLuD/src/SparseEnsembleKalmanInversion.jl:111
| [8] (::EnsembleKalmanProcesses.var"#failsafe_update#26")(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64, Int64}}, u::Matrix{Float64}, g::Matrix{Float64}, y::Matrix{Float64}, obs_noise_cov::Matrix{Float64}, failed_ens::Vector{Int64})
| @ EnsembleKalmanProcesses /central/scratch/esm/slurm-buildkite/calibrateedmf-ci/depot/cpu/packages/EnsembleKalmanProcesses/OjLuD/src/SparseEnsembleKalmanInversion.jl:24
| [9] update_ensemble!(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64, Int64}}, g::Matrix{Float64}; cov_threshold::Float64, Δt_new::Float64, deterministic_forward_map::Bool, failed_ens::Nothing)
| @ EnsembleKalmanProcesses /central/scratch/esm/slurm-buildkite/calibrateedmf-ci/depot/cpu/packages/EnsembleKalmanProcesses/OjLuD/src/SparseEnsembleKalmanInversion.jl:191
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/SparseEnsembleKalmanInversion.jl at line 111 and reproduce the SingularException with the configuration and CEDMF integration test described in the issue. Determine how the SparseInversion update should handle this case, and document how to choose reg and make the convex optimization update robust.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100