stan-dev / stan-dev/cmdstanr

Support compression in read_cmdstan_csv

Open
#1,027 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
R
Stars
160
Forks
69
Avg merge
1d 19h
Merged PRs (30d)
15

Description

Cmdstan output files can get pretty large. data.table::fread used in read_cmdstan_csv and read_csv_metadata supports gzip and bzip2 compressed files if the R.utils package is installed.

On the cmdstanr side this would require at least modifying the suffix assertions (assert_file_exists) in both functions. And, switching out grep in favor of zgrep to handle the commented headers with the caveat (at least on linux) that the argument order be switched so that --color=never comes before the regex pattern.

So in read_cmdstan_csv:

fread_cmd <- paste0("zgrep --color=never -v '^#' '", path.expand(output_file), "'")

read_metadata_csv:

fread_cmd <- paste0("zgrep --color=never '^[#a-zA-Z]' '", path.expand(csv_file), "'")

and similarly for the WSL specific code.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the read_cmdstan_csv and read_csv_metadata entry points, including their WSL-specific code, and inspect the existing assert_file_exists and grep command construction. Support gzip and bzip2 paths while preserving commented-header handling, then verify both functions work across the described command variants.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
cli, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.