dfm / dfm/emcee

HDFBackend not passing compress option

Open
#569 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.6k
Forks
440
PR merge metrics
No merged PRs in 30d

Description

I'm running a set of chains and saving them to HDF using HDFBackend, but it writes out gigantic (many Gb) files which can be externally compressed with gzip to tens of Mb.

I'd like emcee to write out compressed HDF files and I see that the code has a compressed option but it doesn't seem to be linked in the emcee code to the HDF routine.

I'm writing reproducible papers with showyourwork! and I'd like to store just the end set of walkers from a completed emcee run, so if HDF isn't working, I'd rather used a gzipped pickle approach instead - what objects should I store in a pickle in addition to the walker chains, or is there a transparent way to write compressed HDF files using the backend?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading HDFBackend and tracing how its compressed option reaches the HDF routine. Verify the current behavior and identify the relevant backend tests; the work is done when the option produces compressed HDF output or the supported limitation is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.