Add simulation based calibration tooling
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 5
- Forks
- 2
- Avg merge
- 2d 6m
- Merged PRs (30d)
- 7
Description
Goal
The goal here is to add simulation-based calibration model checking to the forecasttools toolbox. This then provides a set of checks for the self-consistency/correctness of a Bayesian model + a posterior sampling method. This is aimed as acting as a unit test for models.
Proposed solution
My proposal is to smooth any issue with using SBC to work with tidybayes (this is likely already available) objects that are structured over multiple sbc trials. In reality, most models we will try outputing an Arviz object so the idea will be to leverage Arviz to tidybayes conversions we already have in place.
At least as a first pass, the user will be responsible for generating the sbc trials.
As a diagram the expected eventual workflow is as follows:
graph TD
Generate_data[Generate datasets from model] -->|inference| posterior_draws[Posterior draws for each dataset]
posterior_draws -->|Arviz| make_data_set[Make InferenceData object over datasets]
make_data_set -->|Serialise| serialise[Serialize to netCDF]
make_data_set -->|direct convert| convert[Convert to tidybayes]
serialise -->|deserialise and convert| convert
convert -->|SBC| sbc[Run SBC checks and return report]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing Arviz-to-tidybayes conversion code in forecasttools and reviewing how tidybayes objects are structured across SBC trials. Compare that path with the SBC workflow described in the issue; done would mean accepting user-generated trials and returning simulation-based calibration checks or a report, but no file or test paths are identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100