Let fits compress their own output CSVs
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 160
- Forks
- 69
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 15
Description
#1027/#1217 make read_cmdstan_csv() and as_cmdstan_fit() read gzip and bzip2 compressed CSVs, but there's no good way to get a fit object's own files compressed. If you gzip the files yourself, the fitted model object still points at the plain .csv paths and fit$draws() will break. When #1217 is merged, the advice for users is basically "finish with the fit, compress, then rebuild it with as_cmdstan_fit()", which is awkward.
But we could add a compress = c("none", "gzip", "bzip2") argument in two places to make this easier for users:
- The fitting methods (
$sample(),$optimize(), etc.), alongsideoutput_dirandoutput_basename. Once CmdStan finishes, cmdstanr compresses the output files in place withgzfile()/bzfile()and records the new paths, so the fit works as usual with no extra call. I guess latent dynamics files should be compressed too whensave_latent_dynamics = TRUE. We can probably skip small files (profile, metric, config). $save_output_files()should support compressing after the fact.
This would break anything that gives file paths to a CmdStan binary like $cmdstan_summary(), $cmdstan_diagnose(), generate_quantities(fitted_params = fit), and laplace(mode = fit). We could either error in those cases or we could decompress to a temp file when needed. Or maybe there's another option. Something to think about.
A few other things that aren't ideal, but can just be documented clearly so users are aware:
- CmdStanMCMC reads the sampler diagnostics right after sampling to print warnings, so that first read requires a decompression. We could avoid by compressing after the fit is build (instead of right after
run_cmdstan()) but that's a more complicated implementation and I think we should keep it simpler. - Every read of a compressed file is slower than the plain CSV. Just need to document this so users keep in mind the tradeoff.
I think reading in the compressed files needs gzip/bzip2 on the PATH (from RTools on Windows)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the fitting methods such as $sample() and $optimize(), along with $save_output_files(), and review how read_cmdstan_csv() and as_cmdstan_fit() store and consume paths. Define how the compress option should update fit-owned files, including latent dynamics, and how commands such as $cmdstan_summary(), $cmdstan_diagnose(), generate_quantities(), and laplace() handle compressed paths. Done should include working gzip and bzip2 output plus clear handling or documentation for affected commands and performance requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100