Diagnostics-driven sampling termination (feature request & design discussion)
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 160
- Forks
- 69
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 15
Description
At present, sampling is terminated when a criterion number of post-warmup samples have been obtained. Given implementation of during-sampling diagnostics, it would be possible to change this termination rule to something more aligned with the user's actual interests, for example:
-
terminating if a post-warmup divergence is encountered
-
terminating when Rhats for all (or a subset) of parameters are below some criterion
-
terminating when Bulk_ESS for all (or a subset) of parameters are above some criterion
-
terminating when Tail_ESS for all (or a subset) of parametersare above some criterion
etc (inc. combinations of criteria)
To achieve this, I propose that chains begin with the requested warmup, but then start sampling runs with a very high value to iter_sampling (does it accept Inf?), then watch the diagnostics as it goes, killing the sampling process when the requested termination criteria have been met.
If warmup/sampling is done as a background process, it would be necessary to spawn another background process for computing the diagnostics and sending the kill signal.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading issue #425 on during-sampling diagnostics and issue #424 on background sampling. Review how iter_sampling is passed to the sampling process and how that process could be stopped. The design is complete when termination criteria, combinations, and process coordination are specified well enough to implement and test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100