stan-dev / stan-dev/cmdstan

"thin" argument affects diagnostics

Open
#1,185 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C++
Stars
238
Forks
106
Avg merge
2h 29m
Merged PRs (30d)
6

Description

Describe the bug
If I use the same seed (to ensure reproducibility) and do two model$fit() runs with identical arguments except that one has thin=4, I get fewer divergent transitions, fewer max_treedepth transitions, and worse EBFMI reported for the run that has thinning. (Probably an underlying Stan issue, not just cmdstanr)

To Reproduce

model <- cmdstanr::cmdstan_model(path_to_funnel_model)

fit1 <- model$sample(data=list(), seed=12345678, chains=1, iter_warmup=1000, iter_sampling=4000)
fit1$diagnostic_summary()
# ^^^ num_divergent = 4, num_max_treedepth = 2, ebfmi = 0.08203262

fit2 <- model$sample(data=list(), seed=12345678, chains=1, iter_warmup=1000, iter_sampling=4000, thin=4)
fit2$diagnostic_summary()
# ^^^ num_divergent = 0, num_max_treedepth = 1, ebfmi = 0.2993997

Expected behavior
Thinning should only affect the number of draws saved, not the diagnostics. A divergent transition is an indication of a possible problem whether or not you save it.

Operating system
Your operating system (e.g. mac os x 10.15, windows 10, etc.)

CmdStanR version number
Your CmdStanR version number (e.g. from packageVersion("cmdstanr")).

Additional context
cmdstanr version 0.6.0.9000
R version 4.2.2
macOS 13.4 (Ventura)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the two model$sample reproductions with identical seeds and compare their diagnostic_summary() results. Trace how thin is handled between sampling and diagnostics, then verify that divergent transitions, max_treedepth transitions, and EBFMI are unchanged except for the saved draw count.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.