Dense Adaptation Fragile

Summary:

The covariance estimation currently used to construct a dense mass matrix is yielding sufficiently poor results that the resulting chains go crazy.

Description:

It looks like the regularization in covar_adaptation.hpp is insufficient. The initial estimates of the covariance are so noisy that they cause the subsequent adaptation windows to fail miserably. Adding stronger regularization helps, but it's not clear if this is a sufficiently robust solution.

Reproducible Steps:

Try to fit the simple model

parameters {
  real mu[10];
}
model {
  mu ~ normal(0, 1);
}

with the CmdStan configuration

./gauss sample num_samples=10000 algorithm=hmc metric=dense_e

Current Output:

Completely biased results. The diagnostics do indicate that the fit results should not be trusted.

Expected Output:

Accurate results. We should be able to fit an IID Gaussian.

Current Version:

develop