Arrow support
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 171
- Forks
- 26
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 3
Description
For a very large number of variables stored in csv, it could be useful to use Arrow to read posterior csv as Arrow data table and use that to let the user to select which variables are actually read to the memory (could be used also for thinning). arrow_table supports dplyr so the implementation of selection and filtering would be relatively easy. It might be easier to just allow this when first time reading the draws from csv, as adding yet another draws type (e.g. draws_arrow_table) would be more work.
Arrow R cheatsheet shows an example of using dplyr
https://github.com/apache/arrow/blob/master/r/cheatsheet/arrow-cheatsheet.pdf
The cheatsheet talks about larger than memory, but I assume it could be faster to not read whole big csv to memory even if it would fit.
I hope the Stan's special comments in csv's are not making this impossible.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no source files or tests. Start by tracing the first-time CSV draw-reading entry point and how Stan CSV special comments are handled, then review the linked Arrow R cheatsheet. Done should include a decided approach for selecting or thinning variables before loading them into memory, with coverage for the supported CSV format.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100