stan-dev / stan-dev/posterior

Arrow support

Open
#240 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

efficiency feature
Dominant language
R
Stars
171
Forks
26
Avg merge
2d 18h
Merged PRs (30d)
3

Description

For a very large number of variables stored in csv, it could be useful to use Arrow to read posterior csv as Arrow data table and use that to let the user to select which variables are actually read to the memory (could be used also for thinning). arrow_table supports dplyr so the implementation of selection and filtering would be relatively easy. It might be easier to just allow this when first time reading the draws from csv, as adding yet another draws type (e.g. draws_arrow_table) would be more work.

Arrow R cheatsheet shows an example of using dplyr
https://github.com/apache/arrow/blob/master/r/cheatsheet/arrow-cheatsheet.pdf
The cheatsheet talks about larger than memory, but I assume it could be faster to not read whole big csv to memory even if it would fit.

I hope the Stan's special comments in csv's are not making this impossible.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no source files or tests. Start by tracing the first-time CSV draw-reading entry point and how Stan CSV special comments are handled, then review the linked Arrow R cheatsheet. Done should include a decided approach for selecting or thinning variables before loading them into memory, with coverage for the supported CSV format.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.