CliMA / CliMA/CalibrateEmulateSample.jl

Blueprint for Dimension reducer / Data Processor

Open
#362 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
90
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Proposal

To construct a data processing tool that can unify the data processing of emulators.

Takes in
- Data: input-output pairs, or the EKI object (with i-o extracted internally)
- a schedule of DataProcessing & Dimension reduction

Gives back
- the processed data pairs

## Example
The user could specify `N`-stage processing:
In each stage the data can be processed with an:
```julia
process_schedule = [
("in", DataProcessor1(...)), # first process inputs with processor 1
("out", DataProcessor2(...)), # next process outputs with processor 2
("joint", DataProcessor3(...)), # next jointly process inputs and outputs with processor 3
("out", DataProcessor4(...)), # finally. process outputs with processor 4
]
```

The user then builds emulators on the processed data
```julia
io_pairs_processed = process_data(io_pairs, process_schedule)
# one catch: interfaces must handle that io_pairs_processed may be a different dimension to io_pairs
```
```julia
io_pairs_recovered= reverse_process_data(io_pairs, process_schedule)
# one catch: interfaces must handle that io_pairs_processed may be a different dimension to io_pairs
```

Where `DataProcessor(...)` could be selected from (just input or output)
- PCA
- simple scalings, or regularization
- Nonlinear dim reduction

or (joint input and output)
- data-informed or likelihood informed subspaces
- CCA

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.