CliMA / CliMA/ClimaCore.jl

Non-determinism in examples, traced back to initial implicit residual in `ImplicitEquationJacobian`

Open
#2,228 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Julia
Stars
117
Forks
19
Avg merge
3d 4h
Merged PRs (30d)
41

Description

Found (by luck) in #2225. I've since initialized the data to zeros, to help improve CI determinism, but if after

```julia
∂Yₜ∂Y = FieldMatrix(...)
```

we do

```julia
fill!(parent(∂Yₜ∂Y[ᶜρ_name, ᶠ𝕄_name]), NaN)
fill!(parent(∂Yₜ∂Y[ᶜ𝔼_name, ᶠ𝕄_name]), NaN)
fill!(parent(∂Yₜ∂Y[ᶠ𝕄_name, ᶜρ_name]), NaN)
fill!(parent(∂Yₜ∂Y[ᶠ𝕄_name, ᶜ𝔼_name]), NaN)
fill!(parent(∂Yₜ∂Y[ᶠ𝕄_name, ᶠ𝕄_name]), NaN)
```

then the solution is (sometimes) driven to NaNs. This appears to be a bug because the solution should not depend on the initial state of the residual--it should be (properly) computed before it is used.

Here are two builds where I've observed this (I also observed this locally):

- https://buildkite.com/clima/climacore-ci/builds/5419#019582d3-de2a-490d-a2f1-a05109e36c87
- https://buildkite.com/clima/climacore-ci/builds/5419#01958518-d497-41c1-b2f2-5bacc4da043e

However, I just tried `∂Yₜ∂Y` with NaNs, and the solution was `NaN`-free, so this seems to be non-deterministic.

Running on the CPU, with Julia 1.10, here is a "reproducer": `ENV["TEST_NAME"] = "sphere/held_suarez_rhoe"; using Revise; include("examples/hybrid/driver.jl")`

cc @dennisYatunin

Contributor guide

No contributing guide indexed for this repository

Research direction

Run the Julia 1.10 reproducer with ENV["TEST_NAME"] = "sphere/held_suarez_rhoe" and include("examples/hybrid/driver.jl"). Trace how ImplicitEquationJacobian uses the residual after ∂Yₜ∂Y is created, using the NaN-filled fields as the diagnostic. Done means the solution is deterministic and does not depend on the residual's initial state or become NaN.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.