CliMA / CliMA/EnsembleKalmanProcesses.jl

Strategies for improvement robustness with JLD2

Open
#549 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
125
Forks
24
Avg merge
1d 18h
Merged PRs (30d)
5

Description

## Issue

It seems like using JLD2 can be quite brittle with updating of EKP/Julia/JLD versions. This can be frustrating for users who have for example the priors / data saved in this format.

It would be good to investigate the most brittle parts of this, and improve/overload the saving functionality, so that rebuilding is more robust. Or at least rebuilding by default could fall back to some flattened objects that can be used to extract the data more easily.

This isnt a small piece of work but could be quite helpful. I think particularly as the use-case is often that EKP is used often for longer term storage relative to it's version evolution.

## Possible solution

How this could work:
1. Define an EKP state (containing the internals of current EKP) for the current version `V1` this can rebuild EKP. (Also do this for ParameterDistribution for priors). It should be in a more robust format than the full EKP object.
2. Define a converter between the EKP and the state, for the current EKP.
3. Whenever the state is changed, define a new state and a migration pattern from one version to the next `V1 -> V2 -> V3` as the state is changed over time. (e.g. replacing missing states with default values etc.)
4. Have JLD2 work with only the state and some metadata (e.g. could just be the state version 1,2,3, ... )
5. Define new methods for the JLD2 [`writeas`, `rconvert`, `wconvert`](https://juliaio.github.io/JLD2.jl/dev/customserialization/) etc. So that users do not see the destructure-restructure patterns and can use `@save` and `@load`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the current EKP and ParameterDistribution serialization paths, then read the linked JLD2 customserialization documentation for writeas, rconvert, and wconvert. Define the versioned state and migration scope before implementation. Done means @save and @load use the state and metadata while preserving access to stored priors and data across supported versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.