Switch from zlib to a modern compression standard
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 402
- Forks
- 134
- PR merge metrics
- No merged PRs in 30d
Description
Improve speed, especially for larger cutouts, by changing the default compression algorithm.
Detailed Description
Currently atlite uses compression={"zlib": True, "complevel": 9, "shuffle": True} as the default in cutout_prepare(). zlib is significantly slower than more modern compression standards such as zstd.
Here is a comparison for a Germany-sized 6-month cutout:
| codec | compression time | final size |
|---|---|---|
| default (zlib, complvl 9) | 63.5 s | 234 MB |
| zstd, complvl 3 | 5.62 s | 297 MB |
| zstd, complvl 3, GranularBitRound sd=5, | 8.7s | 155.4 MB |
Difference in size is due to lack of shuffle for zsd in netcdf4, but size can be optimised by quantising if needed. The GranularBitRound above results in <4e-5 relative error (see here)
For large cutouts this can save hours of processing. Many users of atlite may not know about compression algorithms, so it would be good to provide them with a sane default.
Context
When creating large cutouts with the era5-edh module, which streams chunks directly from an online Zarr store (see #507), the default compression algorithm slows down the pipeline significantly, as dask is often waiting for compression to of one chunk to finish before downloading the next one. Switching to zstd unclogs the stream.
Possible Implementation
@maybe_remove_tmpdir
def cutout_prepare(
cutout,
features=None,
tmpdir=None,
data_format="grib",
overwrite=False,
compression={"zstd": True, "complevel": 3},
show_progress=False,
dask_kwargs=None,
monthly_requests=False,
concurrent_requests=False,
):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the cutout_prepare() entry point shown in the issue and trace how its compression argument is passed through cutout creation. Compare the current zlib default with the proposed zstd settings and check the existing preparation behavior; done means large cutouts use the new default without breaking preparation or user-supplied compression options.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100