stan-dev / stan-dev/loo

Using kfold for model selection after splitting by groups

Open
#169 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
157
Forks
38
Avg merge
4d 16h
Merged PRs (30d)
2

Description

Using packages rstanarm and loo to run a logistic regression with four different intercepts and univariate slope hierarchical by individual, then using k-fold leave-one-group-out for model selection (kfold_split_grouped, then kfold). I get the same error when running the kfold command regardless of the model complexity.
Error message:
Fitting K = 60 models distributed over 3 cores
Error in checkForRemoteErrors(val) :
3 nodes produced errors; first error: object 'n_chains' not found
I think it has to do with the number of cores in the kfold command (The above error occurred with 3 cores). If I run it with one core, it works but impossibly slowly. More than one core, and there is a problem.

I updated R and all packages yesterday:
R Session info:
R version 4.0.5 (2021-03-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
other attached packages:
loo_2.4.1 MCMCvis_0.15.1 rstanarm_2.21.1 Rcpp_1.0.6

I have included some ineligant sample code that should reproduce the error. The problem occurs at line 83.
rstanarm_logistic_hier_test.txt

Thanks,
LKS

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached rstanarm_logistic_hier_test.txt sample and line 83, then reproduce the kfold_split_grouped and kfold calls using three cores and one core. Compare the parallel error involving n_chains with the successful single-core run; done means grouped k-fold model selection runs correctly with multiple cores.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.