HDFBackend not passing compress option
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 440
- PR merge metrics
- No merged PRs in 30d
Description
I'm running a set of chains and saving them to HDF using HDFBackend, but it writes out gigantic (many Gb) files which can be externally compressed with gzip to tens of Mb.
I'd like emcee to write out compressed HDF files and I see that the code has a compressed option but it doesn't seem to be linked in the emcee code to the HDF routine.
I'm writing reproducible papers with showyourwork! and I'd like to store just the end set of walkers from a completed emcee run, so if HDF isn't working, I'd rather used a gzipped pickle approach instead - what objects should I store in a pickle in addition to the walker chains, or is there a transparent way to write compressed HDF files using the backend?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading HDFBackend and tracing how its compressed option reaches the HDF routine. Verify the current behavior and identify the relevant backend tests; the work is done when the option produces compressed HDF output or the supported limitation is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100