Non-determinism in examples, traced back to initial implicit residual in `ImplicitEquationJacobian`
- Dominant language
- Julia
- Stars
- 117
- Forks
- 19
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 41
Description
Found (by luck) in #2225. I've since initialized the data to zeros, to help improve CI determinism, but if after
```julia
∂Yₜ∂Y = FieldMatrix(...)
```
we do
```julia
fill!(parent(∂Yₜ∂Y[ᶜρ_name, ᶠ𝕄_name]), NaN)
fill!(parent(∂Yₜ∂Y[ᶜ𝔼_name, ᶠ𝕄_name]), NaN)
fill!(parent(∂Yₜ∂Y[ᶠ𝕄_name, ᶜρ_name]), NaN)
fill!(parent(∂Yₜ∂Y[ᶠ𝕄_name, ᶜ𝔼_name]), NaN)
fill!(parent(∂Yₜ∂Y[ᶠ𝕄_name, ᶠ𝕄_name]), NaN)
```
then the solution is (sometimes) driven to NaNs. This appears to be a bug because the solution should not depend on the initial state of the residual--it should be (properly) computed before it is used.
Here are two builds where I've observed this (I also observed this locally):
- https://buildkite.com/clima/climacore-ci/builds/5419#019582d3-de2a-490d-a2f1-a05109e36c87
- https://buildkite.com/clima/climacore-ci/builds/5419#01958518-d497-41c1-b2f2-5bacc4da043e
However, I just tried `∂Yₜ∂Y` with NaNs, and the solution was `NaN`-free, so this seems to be non-deterministic.
Running on the CPU, with Julia 1.10, here is a "reproducer": `ENV["TEST_NAME"] = "sphere/held_suarez_rhoe"; using Revise; include("examples/hybrid/driver.jl")`
cc @dennisYatunin
Contributor guide
No contributing guide indexed for this repository
Research direction
Run the Julia 1.10 reproducer with ENV["TEST_NAME"] = "sphere/held_suarez_rhoe" and include("examples/hybrid/driver.jl"). Trace how ImplicitEquationJacobian uses the residual after ∂Yₜ∂Y is created, using the NaN-filled fields as the diagnostic. Done means the solution is deterministic and does not depend on the residual's initial state or become NaN.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100