pydata / pydata/xarray

Improved docs for zarr encoding options.

Open
#9,987 9 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-documentation topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

Hi
I'm been trying to set zarr encoding options from xarray. (zarr3)

Figuring out how to do this isn't straightforward. It wasn't too hard to get this working for most zarr compressors, but getting it working for array-to-bytes codecs - ZFPY and PCodec was rather harder. (the 2 ArrayBytesCodecs). It turned out the issue is that array bytes codecs need specifying as serialisers, rather than as compressors in the encoding object.

Anyway - to cut to the chase, I think some better documentation of the format of the encoding object would be useful. - I've not been able to find any, and resorted to source code reading to find the above parameter.

I'm happy to help write this if useful, but could use a pointer for the best place to put the doc. (I'm new to making xarray changes).

should say though - the fact this works at all just a few days after zarr3 release is great!

Thanks

Format strings that seem to be working for me are as follows (arguably maybe the details of codec naming belong more in zarr land, but at least the serializer keyword is as far as I can see a xarray invention, so should be documented in xarray):

For ArrayBytesCodecs:

encoding = {"serializer": numcodecs.zarr3.()}

For ArrayBytesCodecs:
if in numcodecs:
encoding = {"compressor": numcodecs.zarr3.()}

and if native zarr3:
(note different codec name format)
encoding = {"compressor": zarr.codecs.ZstdCodec()}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No documentation file or test is named in the issue. Start by locating the xarray documentation and source handling encoding options, then document the distinction between serializer and compressor entries, including the shown ZFPY, PCodec, and native zarr3 examples; done means users can identify the correct encoding form without reading source code.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.