PyPSA / PyPSA/atlite

Switch from zlib to a modern compression standard

Open
#508 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: needs classification type: enhancement
Dominant language
Python
Stars
402
Forks
134
PR merge metrics
No merged PRs in 30d

Description

Improve speed, especially for larger cutouts, by changing the default compression algorithm.

Detailed Description

Currently atlite uses compression={"zlib": True, "complevel": 9, "shuffle": True} as the default in cutout_prepare(). zlib is significantly slower than more modern compression standards such as zstd.

Here is a comparison for a Germany-sized 6-month cutout:

codec compression time final size
default (zlib, complvl 9) 63.5 s 234 MB
zstd, complvl 3 5.62 s 297 MB
zstd, complvl 3, GranularBitRound sd=5, 8.7s 155.4 MB

Difference in size is due to lack of shuffle for zsd in netcdf4, but size can be optimised by quantising if needed. The GranularBitRound above results in <4e-5 relative error (see here)

For large cutouts this can save hours of processing. Many users of atlite may not know about compression algorithms, so it would be good to provide them with a sane default.

Context

When creating large cutouts with the era5-edh module, which streams chunks directly from an online Zarr store (see #507), the default compression algorithm slows down the pipeline significantly, as dask is often waiting for compression to of one chunk to finish before downloading the next one. Switching to zstd unclogs the stream.

Possible Implementation

@maybe_remove_tmpdir
def cutout_prepare(
    cutout,
    features=None,
    tmpdir=None,
    data_format="grib",
    overwrite=False,
    compression={"zstd": True, "complevel":  3},
    show_progress=False,
    dask_kwargs=None,
    monthly_requests=False,
    concurrent_requests=False,
):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the cutout_prepare() entry point shown in the issue and trace how its compression argument is passed through cutout creation. Compare the current zlib default with the proposed zstd settings and check the existing preparation behavior; done means large cutouts use the new default without breaking preparation or user-supplied compression options.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.