stan-dev / stan-dev/rstan

Improve convergence diagnostic message for NA

Open
#769 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
1.1k
Forks
266
Avg merge
2h 56m
Merged PRs (30d)
1

Description

Summary:

Related to the discussion of warnings of Rhat and ESS being NA at Stan discourse https://discourse.mc-stan.org/t/largest-rhat-is-na-when-one-parameter-is-fixed-to-a-constant/15488/5
and the older discussion of behavior for constant draws
https://discourse.mc-stan.org/t/ess-rhat-behavior-for-chains-numerical-consistent-with-constants/10465

tl;dr for the above: When all draws for a parameter have the same value, we can't know whether it is due to 1) the parameter being fixed, 2) limited floating point accuracy, 3) for a discrete quantity one state has high probability and the chain is not visiting the other states, 4) chains are stuck. Due to the ambiguity rhat and ess functions return in these cases NA.

It would be helpful to separately report if rhat and ess are bad for NA cases and separately report if there NAs.

Now the warnings are

Warning messages:
1: The largest R-hat is NA, indicating chains have not mixed.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#r-hat 
2: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#bulk-ess 
3: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#tail-ess 

They could be like

Warning messages:
1: Some parameters have constant draws which can be due to parameters being constant, limited numerical accuracy or mixing problems. R-hat, Bulk-ESS and Tail-ESS for these parameters will be reported as NA.
See http://mc-stan.org/misc/warnings.html#r-hat 
2: The largest (non NA) R-hat is x.x, indicating chains have not mixed.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#r-hat 
3: The smallest (non-NA) Bulk Effective Samples Size (ESS) is y.y, indicating posterior means and medians may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#bulk-ess 
4: The smallest (non-NA) Tail Effective Samples Size (ESS) is z.z, indicating posterior variances and tail quantiles may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#tail-ess 

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the R code that generates the R-hat and ESS warning messages and identify the existing diagnostic tests around those warnings. Trace how NA and non-NA diagnostic values are handled, then update the messages and tests so constant-draw cases are reported separately while valid R-hat and ESS warnings retain their non-NA wording.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.