What options do I have for `<U#` type data saving in zarr?
Open
Nobody has claimed this yet.
topic-zarr
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
So I tried out xarray today with zarr version 3.0.4, and encountered these scary warnings:
/nix/store/qasysgiacqplrbda5yl65wg7jrs0gcjl-python3-3.12.9-env/lib/python3.12/site-packages/zarr/codecs/vlen_utf8.py:44: UserWarning: The codec `vlen-utf8` is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
return cls(**configuration_parsed)
/nix/store/qasysgiacqplrbda5yl65wg7jrs0gcjl-python3-3.12.9-env/lib/python3.12/site-packages/zarr/core/array.py:3991: UserWarning: The dtype `<U5` is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
meta = AsyncArray._create_metadata_v3(
/nix/store/qasysgiacqplrbda5yl65wg7jrs0gcjl-python3-3.12.9-env/lib/python3.12/site-packages/zarr/api/asynchronous.py:203: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
warnings.warn(
A MWE is:
import xarray as xr
import numpy as np
xr.DataArray(np.array([
"hello",
"world",
])).to_zarr("test_utf8_strings.zarr")
Is <U5 a variable length utf8 type? It shouldn't be... Also, what are my alternatives?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied xarray and NumPy reproducer and the warnings from zarr's vlen_utf8.py and array.py. Compare the reported <U5 and vlen-utf8 behavior with the Zarr format 3 specification and identify the supported string-saving alternatives. Done means the available options and compatibility implications are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100