Test that empirical posterior distributions of parameters remains the same
- Dominant language
- Python
- Stars
- 161
- Forks
- 140
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 138
Description
This is a re-post of an [issue in ert-examples](https://github.com/equinor/ert-internal-examples/issues/116), because testing properties of the update should be done in ERT or IES repositories. When in ERT, it should be part of the tests of analysis module.
We have multiple snapshot tests in ERT, which tests reproducibility which is good.
What we should guarantee long-term (also when changing seeds / breaking reproducibility) is the distributional assumptions after update.
**Original issues**
Currently we are testing for numerical error on `1e-10` and we cannot change seed
https://github.com/equinor/komodo-releases/actions/runs/7256097675/job/19768394334
see e.g.
https://github.com/equinor/ert-internal-examples/blob/7d025bf98faa174a034cdbc751a225159afa0333/tests/test_hm_tutorial.py#L35
The update algorithms are stochastic algorithms, and we should only guarantee the sampling distributions.
Therefore we should be able to change seed without manual checking. We need automatic testing.
One such test is the Two-sample Kolmogorov-Smirnov test https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov%E2%80%93Smirnov_test
This uses an asymptotic result where ensemble size tends to infinity. If we want to make this more robust, we can employ bootstrapping or permutation techniques. This will come at the cost of more complexity. Thus it makes sense to start with the asymptotic results purely.
Note 1. that the test will be flaky, but we should be able to control the flakyness, e.g. to only occur with probability 1e-10 under the H0 assumption of equal distributions.
Note 2. we will test over multiple parameters that posteriors are correlated. The KS-test works under independence. The "control" probability 1e-10 is therefore not exact.
**Approach**
[A PR was started](https://github.com/equinor/ert-internal-examples/pull/117) that would close the original issue. See separation through Sklar's theorem. Suggests the same approach but splitting the work in sub-issues / PRs
- [ ] #6854
- [ ] #6855
Contributor guide
Research direction
Start by reading the tests of the ERT analysis module and the linked tests/test_hm_tutorial.py example, then review the related work in ert-internal-examples PR #117 and sub-issues #6854 and #6855. Done means automated tests check posterior parameter distributions rather than exact seeded snapshots, with documented control of statistical flakiness and correlated parameters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100