pydata / pydata/xarray

to_zarr() does not maintain time encoding when appending to an existing store

Open
#10,639 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-zarr
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

When writing a dataset to a zarr store, the time encoding is correctly stored and retrieved.
However, when appending to an existing store, it is not possible to set the encoding, and the current encoding is not applied to the appended data.

What did you expect to happen?

I expected the encoding to be consistent between the first write and the subsequent ones.

Minimal Complete Verifiable Example
import tempfile
from datetime import datetime, timedelta
import numpy as np
import xarray as xr
import zarr

with tempfile.TemporaryDirectory() as temp_dir:
    storage = temp_dir
    # Test parameters
    base_time = datetime(year=1, month=1, day=1)
    time_dtype = "datetime64[ms]"
    time_unit = "milliseconds since 1970-01-01T00:00:00"

    time_encoding = {
        "units": time_unit,
        "dtype": time_dtype,
    }

    print(f"Base time: {base_time}")
    print(f"Time encoding: {time_encoding}")

    # Write first timestep
    print("\n--- Writing first timestep ---")
    sim_time1 = base_time + timedelta(minutes=1)
    time_coord1 = np.array([np.datetime64(sim_time1, "ms")], dtype=time_dtype)

    data1 = xr.DataArray(
        data=np.array([[[1.0, 2.0], [3.0, 4.0]]]),
        coords={"time": time_coord1, "y": [0, 1], "x": [0, 1]},
        dims=["time", "y", "x"],
        name="test_var"
    )

    ds1 = xr.Dataset({"test_var": data1})

    ds1.to_zarr(
        storage,
        encoding={"time": time_encoding},  # Encoding provided here
        mode="w",
    )

    print(f"Written: {sim_time1}")

    # Write second timestep using append_dim (this is where the bug occurs)
    print("\n--- Writing second timestep with append_dim ---")
    sim_time2 = base_time + timedelta(minutes=2)
    time_coord2 = np.array([np.datetime64(sim_time2, "ms")], dtype=time_dtype)

    data2 = xr.DataArray(
        data=np.array([[[5.0, 6.0], [7.0, 8.0]]]),
        coords={"time": time_coord2, "y": [0, 1], "x": [0, 1]},
        dims=["time", "y", "x"],
        name="test_var"
    )

    ds2 = xr.Dataset({"test_var": data2})

    ds2.to_zarr(
        storage,
        append_dim="time",
        mode="a",
        # NOTE: Cannot pass encoding={"time": time_encoding} here!
        # xarray raises: "variable 'time' already exists, but encoding was provided"
    )

    print(f"Written: {sim_time2}")

    # Read back and demonstrate the bug
    print("\n--- Reading back data ---")
    ds_read = xr.open_zarr(storage)

    print(f"Time coordinate values: {ds_read['time'].values}")
    print(f"Time dtype: {ds_read['time'].dtype}")

    # Expected vs actual
    expected_times = [
        np.datetime64(sim_time1, "ms"),
        np.datetime64(sim_time2, "ms")
    ]
    actual_times = ds_read['time'].values

    print(f"\nExpected: {expected_times}")
    print(f"Actual:   {actual_times}")

    # Check if bug is present
    assert np.array_equal(expected_times, actual_times)
MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
Base time: 0001-01-01 00:00:00
Time encoding: {'units': 'milliseconds since 1970-01-01T00:00:00', 'dtype': 'datetime64[ms]'}

--- Writing first timestep ---
/home/laurent/software/itzi/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:228: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(
Written: 0001-01-01 00:01:00

--- Writing second timestep with append_dim ---
/home/laurent/software/itzi/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:228: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(
Written: 0001-01-01 00:02:00

--- Reading back data ---
Time coordinate values: ['0001-01-01T00:01:00.000' '1970-01-01T00:00:00.000']
Time dtype: datetime64[ms]

Expected: [np.datetime64('0001-01-01T00:01:00.000'), np.datetime64('0001-01-01T00:02:00.000')]
Actual:   ['0001-01-01T00:01:00.000' '1970-01-01T00:00:00.000']
Traceback (most recent call last):
  File "/home/laurent/software/itzi/xarray_zarr_time.py", line 86, in <module>
    assert np.array_equal(expected_times, actual_times)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
python-bits: 64
OS: Linux
OS-release: 6.14.0-27-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.4-development

xarray: 2025.7.1
pandas: 2.3.1
numpy: 2.3.2
scipy: 1.16.1
netCDF4: 1.7.2
pydap: 3.5.5
h5netcdf: 1.6.4
h5py: 3.14.0
zarr: 3.1.1
cftime: 1.6.4.post1
nc_time_axis: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2025.7.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 80.9.0
pip: None
conda: None
pytest: 8.4.1
mypy: None
IPython: None
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the to_zarr append_dim path and the encoding handling used by open_zarr, using the reproducible script xarray_zarr_time.py as the failing example. Confirm that appended time values preserve the first write's encoding and match the expected two datetime64[ms] values without allowing an incompatible encoding override.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.