easystats / easystats/performance
Comparing frequentist and Bayesian models
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 1.2k
- Forks
- 109
- Avg merge
- 6h 34m
- Merged PRs (30d)
- 8
Description
Is it possible to compare a Bayesian and frequentist model in a meaningful way? I tried compare_performance() for comparing the same model specified with brm and glmmTMB, but the error suggests to me that it might not be possible (or meaningful). Error is:
> performance::compare_performance(
+ mod_ratio_state,
+ bayes_ratio_state)
Error in loo::stacking_weights(lpd_point) :
At least two models are required for stacking weights.
(I'm trying to make a case for using Variance Ratio over ICC, and am trying to pre-empt questions about the model fits).
Might relate to https://github.com/easystats/performance/issues/716.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with performance::compare_performance() and reproduce the example using the brm and glmmTMB models. Inspect the loo::stacking_weights error and the related issue #716. Done means establishing whether cross-framework comparison is supported and defining the expected behavior or documentation for this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- analytics, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100