CliMA / CliMA/CalibrateEmulateSample.jl

Add a diagnostic to warn users if the GP training learns a 0 whitekernel

Open
#264 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
90
Forks
16
PR merge metrics
No merged PRs in 30d

Description

It is not uncommon to see output from GP training that looks like this:

_This comes from a `2D -> 2D` problem training ARD kernel with `GaussianProcess(...,noise_learn=true)`, (the default)_

```
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
Type: GaussianProcesses.SEArd{Float64}, Params: [-0.0, -0.0, 0.0] Type: GaussianProcesses.Noise{Float64}, Params: [0.0]
created GP: 1
kernel in GaussianProcess:
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
Type: GaussianProcesses.SEArd{Float64}, Params: [-0.0, -0.0, 0.0] Type: GaussianProcesses.Noise{Float64}, Params: [0.0]
created GP: 2
PosDefException(2)
PosDefException(6)
optimized hyperparameters of GP: 1
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
Type: GaussianProcesses.SEArd{Float64}, Params: [1.5898189312537052, 4.284595166329162, 3.31913265553596] Type: GaussianProcesses.Noise{Float64}, Params: [-25.115891356746516]
PosDefException(5)
optimized hyperparameters of GP: 2
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
Type: GaussianProcesses.SEArd{Float64}, Params: [8.94388355520155, 4.631539162650433, 6.172770146416866] Type: GaussianProcesses.Noise{Float64}, Params: [-17.73352141936848]
```
Paying attention to the `Noise` kernel, it learns values `-25` and `-17` after training, (these are in log space) so the kernel learns a noise of `exp(-25)` and `exp(-3)` is learnt . The default regularization `alg_reg_noise` (=minimum noise) is `10^-3`, so basically we end up with a noise of `10^-3`.

## Solution
- if the solution produces overly confident (spiked) posteriors: increase `alg_reg_noise` with `noise_learn = true` can help regularize the training more
- if the posterior with this result still looks reasonable: then `alg_reg_noise` may be too large, (i.e. the true noise you are trying to learn is less than `alg_reg_noise`)
- set `noise_learn = false` (which sets `alg_reg_noise = 1` (in a normalized space) and doesn't learn the extra kernel) tends to solve the problem too

## Action to be taken

Add a warning to users if the `noise learnt << alg_reg_noise`. Suggest the above solutions.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at GaussianProcess(..., noise_learn=true) and trace how the Noise kernel and alg_reg_noise are handled during training. The issue names no files or tests; done means warning when learned noise is much smaller than alg_reg_noise and suggesting the listed remedies.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.