easystats / easystats/performance
Revise compare_models() for Bayesian models
Open
@strengejacke is already working on this.
Since Nov 24, 2024.
Consistency :green_apple: :apple:
Enhancement :boom:
- Dominant language
- R
- Stars
- 1.2k
- Forks
- 109
- Avg merge
- 6h 34m
- Merged PRs (30d)
- 8
Description
The current output would benefit from some streamlining.
> performance::compare_performance(model, model2, model3)
# Comparison of Model Performance Indices
Name | Model | ELPD | ELPD_SE | LOOIC (weights) | LOOIC_SE | WAIC (weights) | R2 | R2 (adj.) | RMSE | Sigma
-----------------------------------------------------------------------------------------------------------------------
model | brmsfit | -104.387 | 9.532 | 208.8 (0.092) | 19.063 | 208.8 (<.001) | 0.927 | 0.926 | 0.475 | 0.487
model2 | brmsfit | -89.212 | 10.543 | 178.4 (<.001) | 21.085 | 178.4 (<.001) | 0.941 | 0.940 | 0.425 | 0.467
model3 | brmsfit | -64.580 | 11.569 | 129.2 (0.908) | 23.137 | 129.1 (>.999) | 0.958 | 0.957 | 0.353 | 0.363
Suggestions:
- Remove RMSE and Sigma by default (not exactly typically useful or what you would expect when comparing models)
- Move R2 and R2 adj. first, after the model column
- My current understanding is that LOO and WAIC are both methods to estimate ELPD, which can be used to compare models (in particular, by looking at the ELPD-diff (see https://github.com/easystats/report/pull/419 for the report support recently added). As such, in the interest of computation, I would display directly the ELPD_DIFF + (SE), by default computed using LOO (although we could add the option for WAIC for faster computation), and that's it, drop the LOOIC, WAIC and the raw ELPD which are redundant indices and simply add noise.
What do you think?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.