Store samples in matrix rather than dataframe
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 1.1k
- Forks
- 266
- Avg merge
- 2h 56m
- Merged PRs (30d)
- 1
Description
Allocating very wide data.frames can take several hours. Though, data.frames take roughly the same amount of memory as matrices, they are much slower to allocate, create many more objects, and slow down garbage collection.
It would be nice to store the samples as matrices to make things faster. I am have a branch that has read_stan_csv create a matrix rather than a data.frame, while maintaining support for previous data.frame based stan.fit objects.
For a 10,000 parameter model, this is an additional 15% speed improvement in reading in the data over https://github.com/stan-dev/rstan/pull/532, and I believe would offer even greater speed improvements for larger models.
I'm not sure why things were initially created using data frames, but if using matrices makes sense, I can create a pull request.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with read_stan_csv and the stan.fit objects mentioned in the issue, then inspect the use_matrices branch to understand the proposed representation. Check how existing data.frame-based stan.fit objects remain supported and compare sample-reading performance for large models. Done means samples are stored as matrices without breaking previous stan.fit objects.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100