CliMA / CliMA/CalibrateEmulateSample.jl
Add a diagnostic to warn users if the GP training learns a 0 whitekernel
- Dominant language
- Julia
- Stars
- 90
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
It is not uncommon to see output from GP training that looks like this:
_This comes from a `2D -> 2D` problem training ARD kernel with `GaussianProcess(...,noise_learn=true)`, (the default)_
```
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
Type: GaussianProcesses.SEArd{Float64}, Params: [-0.0, -0.0, 0.0] Type: GaussianProcesses.Noise{Float64}, Params: [0.0]
created GP: 1
kernel in GaussianProcess:
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
Type: GaussianProcesses.SEArd{Float64}, Params: [-0.0, -0.0, 0.0] Type: GaussianProcesses.Noise{Float64}, Params: [0.0]
created GP: 2
PosDefException(2)
PosDefException(6)
optimized hyperparameters of GP: 1
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
Type: GaussianProcesses.SEArd{Float64}, Params: [1.5898189312537052, 4.284595166329162, 3.31913265553596] Type: GaussianProcesses.Noise{Float64}, Params: [-25.115891356746516]
PosDefException(5)
optimized hyperparameters of GP: 2
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
Type: GaussianProcesses.SEArd{Float64}, Params: [8.94388355520155, 4.631539162650433, 6.172770146416866] Type: GaussianProcesses.Noise{Float64}, Params: [-17.73352141936848]
```
Paying attention to the `Noise` kernel, it learns values `-25` and `-17` after training, (these are in log space) so the kernel learns a noise of `exp(-25)` and `exp(-3)` is learnt . The default regularization `alg_reg_noise` (=minimum noise) is `10^-3`, so basically we end up with a noise of `10^-3`.
## Solution
- if the solution produces overly confident (spiked) posteriors: increase `alg_reg_noise` with `noise_learn = true` can help regularize the training more
- if the posterior with this result still looks reasonable: then `alg_reg_noise` may be too large, (i.e. the true noise you are trying to learn is less than `alg_reg_noise`)
- set `noise_learn = false` (which sets `alg_reg_noise = 1` (in a normalized space) and doesn't learn the extra kernel) tends to solve the problem too
## Action to be taken
Add a warning to users if the `noise learnt << alg_reg_noise`. Suggest the above solutions.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at GaussianProcess(..., noise_learn=true) and trace how the Noise kernel and alg_reg_noise are handled during training. The issue names no files or tests; done means warning when learned noise is much smaller than alg_reg_noise and suggesting the listed remedies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100