stan-dev / stan-dev/cmdstanr

Diagnostics-driven sampling termination (feature request & design discussion)

Open
#426 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
R
Stars
160
Forks
69
Avg merge
1d 19h
Merged PRs (30d)
15

Description

At present, sampling is terminated when a criterion number of post-warmup samples have been obtained. Given implementation of during-sampling diagnostics, it would be possible to change this termination rule to something more aligned with the user's actual interests, for example:

  • terminating if a post-warmup divergence is encountered

  • terminating when Rhats for all (or a subset) of parameters are below some criterion

  • terminating when Bulk_ESS for all (or a subset) of parameters are above some criterion

  • terminating when Tail_ESS for all (or a subset) of parametersare above some criterion

etc (inc. combinations of criteria)

To achieve this, I propose that chains begin with the requested warmup, but then start sampling runs with a very high value to iter_sampling (does it accept Inf?), then watch the diagnostics as it goes, killing the sampling process when the requested termination criteria have been met.

If warmup/sampling is done as a background process, it would be necessary to spawn another background process for computing the diagnostics and sending the kill signal.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading issue #425 on during-sampling diagnostics and issue #424 on background sampling. Review how iter_sampling is passed to the sampling process and how that process could be stopped. The design is complete when termination criteria, combinations, and process coordination are specified well enough to implement and test.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.