pydata / pydata/xarray

What options do I have for `<U#` type data saving in zarr?

Open
#10,077 8 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

So I tried out xarray today with zarr version 3.0.4, and encountered these scary warnings:

/nix/store/qasysgiacqplrbda5yl65wg7jrs0gcjl-python3-3.12.9-env/lib/python3.12/site-packages/zarr/codecs/vlen_utf8.py:44: UserWarning: The codec `vlen-utf8` is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  return cls(**configuration_parsed)
/nix/store/qasysgiacqplrbda5yl65wg7jrs0gcjl-python3-3.12.9-env/lib/python3.12/site-packages/zarr/core/array.py:3991: UserWarning: The dtype `<U5` is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  meta = AsyncArray._create_metadata_v3(
/nix/store/qasysgiacqplrbda5yl65wg7jrs0gcjl-python3-3.12.9-env/lib/python3.12/site-packages/zarr/api/asynchronous.py:203: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(

A MWE is:

import xarray as xr
import numpy as np

xr.DataArray(np.array([
    "hello",
    "world",
])).to_zarr("test_utf8_strings.zarr")

Is <U5 a variable length utf8 type? It shouldn't be... Also, what are my alternatives?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied xarray and NumPy reproducer and the warnings from zarr's vlen_utf8.py and array.py. Compare the reported <U5 and vlen-utf8 behavior with the Zarr format 3 specification and identify the supported string-saving alternatives. Done means the available options and compatibility implications are clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.