Derive proper correlation matrix when resampling profiles
- Dominant language
- Python
- Stars
- 4
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
To avoid the duplication of critical code related to the propagation of uncertainties, the resampling of profiles --which is implemented as a strategy of the `MultiProfile` classes in `dvas.data.strategy.resample.py` -- uses the function `process_chunks` from `dvas.tools.gdps.utils.py`.
To do so, the resampling step "mimics" a delta by computing two suitable "profiles", the difference of which corresponds to the resample target. Namely, the first profile corresponds to `x_- * (omega-1)`, while the second is `x_+ * omega`. When doing second-first (which is the chosen convention of the `delta` mode of `process_chunks`), I get `x_- * (1-omega) + x_+ * omega`, which is the linearly resampled values I am looking for.
**However:**
The correlations coefficients computed via the `coeffs()` function assume that we are combining two profiles that have been synchronized, such that the index `i` and `j` are derived simply from the row number. In the case of the resampling, this is incorrect, as the second "profile" really is just a distinct set of neighboring points from the same unique profile. As such, both profiles share the same `oid`, `rid` and `mid`, and should only different in their `i` and `j` values. But since `j` and `i` are derived directly by `coeffs()`, there is currently no way to set them differently when doing a resampling.
As a work-around, I am instead setting a distinct `oid` as of #199. This works for now, and ensures a correct estimations of the uncorrelated errors, **but only because the `oid` value plays no role in any of the other existing types of correlations**.
Most likely, this will remain so for the foreseeable future, and this bug will thus not be worked on. It is kept here for legacy purposes, should the situation ever change and warrant a correction.
It should also be noted that the resampling step currently also ignore the correlation of the interpolated values resulting from the underlying use of the same anchors.
I guess the point to take home is that resampling with dvas really should be done **as an absolutly-last resort measure**.
Contributor guide
Research direction
Start with the resampling strategy in dvas.data.strategy.resample.py and the process_chunks and coeffs() implementations referenced in dvas.tools.gdps.utils.py. Determine how resampled profiles should provide distinct i and j values and how correlations from shared interpolation anchors should be represented; done means resampling derives the proper correlation matrix without relying on a distinct oid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100