Improve convergence diagnostic message for NA
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 1.1k
- Forks
- 266
- Avg merge
- 2h 56m
- Merged PRs (30d)
- 1
Description
Summary:
Related to the discussion of warnings of Rhat and ESS being NA at Stan discourse https://discourse.mc-stan.org/t/largest-rhat-is-na-when-one-parameter-is-fixed-to-a-constant/15488/5
and the older discussion of behavior for constant draws
https://discourse.mc-stan.org/t/ess-rhat-behavior-for-chains-numerical-consistent-with-constants/10465
tl;dr for the above: When all draws for a parameter have the same value, we can't know whether it is due to 1) the parameter being fixed, 2) limited floating point accuracy, 3) for a discrete quantity one state has high probability and the chain is not visiting the other states, 4) chains are stuck. Due to the ambiguity rhat and ess functions return in these cases NA.
It would be helpful to separately report if rhat and ess are bad for NA cases and separately report if there NAs.
Now the warnings are
Warning messages:
1: The largest R-hat is NA, indicating chains have not mixed.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#r-hat
2: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#bulk-ess
3: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#tail-ess
They could be like
Warning messages:
1: Some parameters have constant draws which can be due to parameters being constant, limited numerical accuracy or mixing problems. R-hat, Bulk-ESS and Tail-ESS for these parameters will be reported as NA.
See http://mc-stan.org/misc/warnings.html#r-hat
2: The largest (non NA) R-hat is x.x, indicating chains have not mixed.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#r-hat
3: The smallest (non-NA) Bulk Effective Samples Size (ESS) is y.y, indicating posterior means and medians may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#bulk-ess
4: The smallest (non-NA) Tail Effective Samples Size (ESS) is z.z, indicating posterior variances and tail quantiles may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#tail-ess
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the R code that generates the R-hat and ESS warning messages and identify the existing diagnostic tests around those warnings. Trace how NA and non-NA diagnostic values are handled, then update the messages and tests so constant-draw cases are reported separately while valid R-hat and ESS warnings retain their non-NA wording.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100