CliMA / CliMA/EnsembleKalmanProcesses.jl
Strategies for improvement robustness with JLD2
- Dominant language
- Julia
- Stars
- 125
- Forks
- 24
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 5
Description
## Issue
It seems like using JLD2 can be quite brittle with updating of EKP/Julia/JLD versions. This can be frustrating for users who have for example the priors / data saved in this format.
It would be good to investigate the most brittle parts of this, and improve/overload the saving functionality, so that rebuilding is more robust. Or at least rebuilding by default could fall back to some flattened objects that can be used to extract the data more easily.
This isnt a small piece of work but could be quite helpful. I think particularly as the use-case is often that EKP is used often for longer term storage relative to it's version evolution.
## Possible solution
How this could work:
1. Define an EKP state (containing the internals of current EKP) for the current version `V1` this can rebuild EKP. (Also do this for ParameterDistribution for priors). It should be in a more robust format than the full EKP object.
2. Define a converter between the EKP and the state, for the current EKP.
3. Whenever the state is changed, define a new state and a migration pattern from one version to the next `V1 -> V2 -> V3` as the state is changed over time. (e.g. replacing missing states with default values etc.)
4. Have JLD2 work with only the state and some metadata (e.g. could just be the state version 1,2,3, ... )
5. Define new methods for the JLD2 [`writeas`, `rconvert`, `wconvert`](https://juliaio.github.io/JLD2.jl/dev/customserialization/) etc. So that users do not see the destructure-restructure patterns and can use `@save` and `@load`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the current EKP and ParameterDistribution serialization paths, then read the linked JLD2 customserialization documentation for writeas, rconvert, and wconvert. Define the versioned state and migration scope before implementation. Done means @save and @load use the state and metadata while preserving access to stored priors and data across supported versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100