pydata / pydata/xarray

ds.to_zarr() fails when trying to write a very large zarr of dtype='object', even with compute=False

Open
#10,583 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs info topic-dask topic-performance topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

When making an empty zarr full of object dtypes, I got a MemoryError, despite setting compute=False.

What did I expect to happen?
I expected to save the empty zarr to disk.
My use case is that I am trying to create a zarr dataset which will be populated with variable length utf-8 strings, which I know was not previously supported with zarr, but it now is. The issue, I think, can be explained by this comment in zarr.py:

File ~/miniconda3/lib/python3.13/site-packages/xarray/backends/zarr.py:531, in encode_zarr_variable(var, needs_copy, name)
    510 """
    511 Converts an Variable into an Variable which follows some
    512 of the CF conventions:
   (...)    527     A variable which has been encoded as described above.
    528 """
    530 var = conventions.encode_cf_variable(var, name=name)
--> 531 var = ensure_dtype_not_object(var, name=name)
    533 # zarr allows unicode, but not variable-length strings, so it's both
    534 # simpler and more compact to always encode as UTF-8 explicitly.
    535 # TODO: allow toggling this explicitly via dtype in encoding.
    536 # TODO: revisit this now that Zarr _does_ allow variable-length strings
    537 coder = coding.strings.EncodedStringCoder(allows_unicode=True)

MCVE:

dummies = dask.array.zeros((5000, 100, 2000, 50), chunks=(10, 10, 500, 50), dtype = np.dtypes.StringDType)
ds = xr.Dataset({"foo": (["x", "y", "z", "alpha"], dummies)}, coords={"x": np.arange(5000), "y" : np.arange(100), "z" : np.arange(2000), "alpha" : np.arange(50)})
bigZarr = xr.merge([ds,dsf])
bigZarr.to_zarr('myzarr.zarr', compute=False, consolidated=False)
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.

  • Complete example — the example is self-contained, including all data and the text of any traceback.

  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.

  • New issue — a search of GitHub Issues suggests this is not a duplicate.

  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Also, in the mean time, if anyone has a recommendation for how to make my project work regardless of this limitation, I would be keen to hear how. Much appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in xarray/backends/zarr.py at encode_zarr_variable(), especially the call to ensure_dtype_not_object(), and run the reported MCVE with compute=False. Trace why object-dtype allocation occurs during encoding rather than deferred writing. Done means the large empty object-dtype dataset can be written to Zarr without a MemoryError while retaining support for the stated variable-length UTF-8 use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.