equinor / equinor/ert

Test that empirical posterior distributions of parameters remains the same

Open
#6,853 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
161
Forks
140
Avg merge
2d 1h
Merged PRs (30d)
138

Description

This is a re-post of an [issue in ert-examples](https://github.com/equinor/ert-internal-examples/issues/116), because testing properties of the update should be done in ERT or IES repositories. When in ERT, it should be part of the tests of analysis module.

We have multiple snapshot tests in ERT, which tests reproducibility which is good.
What we should guarantee long-term (also when changing seeds / breaking reproducibility) is the distributional assumptions after update.

**Original issues**
Currently we are testing for numerical error on `1e-10` and we cannot change seed
https://github.com/equinor/komodo-releases/actions/runs/7256097675/job/19768394334
see e.g.
https://github.com/equinor/ert-internal-examples/blob/7d025bf98faa174a034cdbc751a225159afa0333/tests/test_hm_tutorial.py#L35

The update algorithms are stochastic algorithms, and we should only guarantee the sampling distributions.
Therefore we should be able to change seed without manual checking. We need automatic testing.

One such test is the Two-sample Kolmogorov-Smirnov test https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov%E2%80%93Smirnov_test

This uses an asymptotic result where ensemble size tends to infinity. If we want to make this more robust, we can employ bootstrapping or permutation techniques. This will come at the cost of more complexity. Thus it makes sense to start with the asymptotic results purely.

Note 1. that the test will be flaky, but we should be able to control the flakyness, e.g. to only occur with probability 1e-10 under the H0 assumption of equal distributions.

Note 2. we will test over multiple parameters that posteriors are correlated. The KS-test works under independence. The "control" probability 1e-10 is therefore not exact.

**Approach**
[A PR was started](https://github.com/equinor/ert-internal-examples/pull/117) that would close the original issue. See separation through Sklar's theorem. Suggests the same approach but splitting the work in sub-issues / PRs
- [ ] #6854
- [ ] #6855

Contributor guide

Open the contributing guide

Research direction

Start by reading the tests of the ERT analysis module and the linked tests/test_hm_tutorial.py example, then review the related work in ert-internal-examples PR #117 and sub-issues #6854 and #6855. Done means automated tests check posterior parameter distributions rather than exact seeded snapshots, with documented control of statistical flakiness and correlated parameters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.