Strings in coordinates may be truncated when saving concatenated rasters to zarr

What happened?

I would like to concatenate two dataarrays or two datasets, save them to zarr, read them into xarray again at a later time, and have all the coordinates intact.

However, if I concatenate two dataarrays or datasets along a string-based coordinate, if the first dataarray in the concatenation has a coordinate with shorter maximum string length than the second dataarray, saving the concatenated dataset to zarr can truncate the string.

This problem only arises when:

1. First dataarray's string coordinate has an encoding["dtype"] attribute
2. First dataarray's string coordinate length is shorter than the second dataarray's.

What did you expect to happen?

I expect the xarray dataset that was read from the zarr to have the string coordinate the same as the input dataset that I wrote in, with no truncation.

Minimal Complete Verifiable Example

import logging
import tempfile
from pathlib import Path

import dask.array as da
import numpy as np
import xarray as xr

logger = logging.getLogger(__name__)


def mb_raster_with_custom_band_names(bands: list[str]) -> xr.DataArray:
    """
    Generates a multi-band raster with custom band names.

    This function creates a multi-band raster with a specified list of band names. The raster is generated with random
    data and has a shape of (len(bands), 100, 100). The raster is assigned the EPSG:32733 coordinate reference system.

    :param bands: A list of strings specifying the band names for the raster.
    :return: A multi-band raster with custom band names as an xarray DataArray.
    """
    n_x = 100
    n_y = 100

    data_with_custom_band_names = xr.DataArray(
        (da.random.random_sample(size=(len(bands), n_x, n_y)) * 255).astype("float32"),
        coords={"band": bands, "y": np.arange(n_y), "x": np.arange(n_x)},
    )
    return data_with_custom_band_names


def zarr_round_trip(raster: xr.DataArray | xr.Dataset) -> xr.DataArray:
    """Simple function to test the round trip of a raster to zarr and back."""
    with tempfile.TemporaryDirectory() as tmpdirname:
        tmp_zarr_path = Path(tmpdirname) / "tmp_zarr.zarr"
        raster.to_zarr(tmp_zarr_path, mode="w")

        round_trip_raster = xr.open_dataset(tmp_zarr_path, engine="zarr")

    return round_trip_raster


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)

    # Make a multi-band raster with short band names.
    rgb_raster = mb_raster_with_custom_band_names(["blue", "green", "red"])
    # Force the band encoding dtype to be equal to the band value dtype. This is what was triggering the band name
    # truncation issue when saving to zarr.
    rgb_raster.band.encoding["dtype"] = rgb_raster.band.dtype

    # Make another multi-band raster with long band names.
    long_rgb_raster = mb_raster_with_custom_band_names(["the_sky_is_blue", "the_grass_is_green", "the_rose_is_red"])
    long_rgb_raster.band.encoding["dtype"] = long_rgb_raster.band.dtype

    # concat two rasters with xr_concat (short band name raster first)
    concat_raster_short_first = xr.concat([rgb_raster, long_rgb_raster], dim="band")

    logger.info(f"{concat_raster_short_first.band.values=}")
    # array(['blue', 'green', 'red', 'the_sky_is_blue', 'the_grass_is_green', 'the_rose_is_red'], dtype='<U18')

    logger.info(f"{concat_raster_short_first.band.dtype=}")
    # dtype('<U18')

    logger.info(f"{concat_raster_short_first.band.encoding['dtype']=}")
    # dtype('<U5')

    concat_raster_short_first_reread = zarr_round_trip(concat_raster_short_first)
    logger.info(f"{concat_raster_short_first_reread.band.values=}")
    # array(['blue', 'green', 'red', 'the_s', 'the_g', 'the_r'], dtype='<U5')

    logger.info(f"{concat_raster_short_first_reread.band.dtype=}")
    # dtype('<U5')

    logger.info(f"{concat_raster_short_first_reread.band.encoding['dtype']=}")
    # dtype('<U5')

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

INFO:__main__:concat_raster_short_first.band.values=array(['blue', 'green', 'red', 'the_sky_is_blue', 'the_grass_is_green',
       'the_rose_is_red'], dtype='<U18')
INFO:__main__:concat_raster_short_first.band.dtype=dtype('<U18')
INFO:__main__:concat_raster_short_first.band.encoding['dtype']=dtype('<U5')
INFO:__main__:concat_raster_short_first_reread.band.values=array(['blue', 'green', 'red', 'the_s', 'the_g', 'the_r'], dtype='<U5')
INFO:__main__:concat_raster_short_first_reread.band.dtype=dtype('<U5')
INFO:__main__:concat_raster_short_first_reread.band.encoding['dtype']=dtype('<U5')

Anything else we need to know?

The order of the dataarray in xr.concat mattered. If you run xr.concat([rgb_raster, long_rgb_raster], dim="band"), the truncation will happen; but if you run xr.concat([long_rgb_raster, rgb_raster], dim="band") , it will not.

I attempted to unwrap the call chain when we save a dataset or dataarray to zarr:

• In the xarray.DataArray.to_zarr() function, xarray.DataArray is implicitly converted to xarray.Dataset before we save it to zarr: code.
• Because we’re calling xarray.DataArray.to_dataset() directly with no additional argument, it calls xarray.DataArray._to_dataset_whole().
• In xarray.DataArray._to_dataset_whole(), the coordinates attribute (a private attr called ._coords) of the dataarray is implicitly copied to the variable of the dataset. So whatever was in our dataset variables, it came from the ._coords private attr of the datarray.
• The next clue as to how the dataset variables get saved to zarr is in xarray.backends.ZarrStore. Specifically, the dataset variable go through this encode_zarr_variable function. That function in turn calls xarray.conventions.encode_cf_variable. In there, the function just uses whatever the variable encoding was in the dataset, which came from the coordinate encoding from the data array.

Looks like a similar issue was handled in 2014 (https://github.com/pydata/xarray/issues/217); this bug appears to a corner case of when the encoding attr of a particular coordinate was left intact during concatenation.

Environment

Reproduce the provided concatenation and zarr round trip first, then trace the coordinate encoding through xarray/core/dataarray.py, xarray/backends/zarr.py, and xarray/conventions.py. Add a regression test using the short-name-first case; done means reopening the zarr preserves every full string coordinate without truncation.