default compression for embedded data
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Currently, any embedded arrays are written to the output by zarr using default options. This means it will be blosc-compressed, which has downsides:
- the user needs to have blosc available
- for small arrays, the extra header and frame likely means bigger output
- the final output should be compressed anyway (Zstd works well for the large number of strings we usually encounter, or of course whatever would work well in parquet if we went in that direction)
Should we set the compression to None or something else?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating where embedded arrays are serialized with zarr's default options; the issue does not name a file or test. Compare the output and dependency implications of the proposed compression choices, then confirm the intended default with maintainers. Done means an agreed compression behavior is implemented and covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100