Appending to zarr with string dtype

import xarray as xr

da = xr.DataArray(['foo'])
ds = da.to_dataset(name='da')
ds.to_zarr('ds') # no special encoding specified

ds = xr.open_zarr('ds')
print(ds.da.values)

The following code prints ['foo'] (string type). The encoding chosen by zarr is "dtype": "|S3", which corresponds to bytes, but it seems to be decoded to a string, which is what we want.

$ cat ds/da/.zarray 
{
    "chunks": [
        1
    ],
    "compressor": {
        "blocksize": 0,
        "clevel": 5,
        "cname": "lz4",
        "id": "blosc",
        "shuffle": 1
    },
    "dtype": "|S3",
    "fill_value": null,
    "filters": null,
    "order": "C",
    "shape": [
        1
    ],
    "zarr_format": 2
}

The problem is that if I want to append to the zarr archive, like so:

import zarr

ds = zarr.open('ds', mode='a')
da_new = xr.DataArray(['barbar'])
ds.da.append(da_new)

ds = xr.open_zarr('ds')
print(ds.da.values)

It prints ['foo' 'bar']. Indeed the encoding was kept as "dtype": "|S3", which is fine for a string of 3 characters but not for 6.

If I want to specify the encoding with the maximum length, e.g:

ds.to_zarr('ds', encoding={'da': {'dtype': '|S6'}})

It solves the length problem, but now my strings are kept as bytes: [b'foo' b'barbar']. If I specify a Unicode encoding:

ds.to_zarr('ds', encoding={'da': {'dtype': 'U6'}})

It is not taken into account. The zarr encoding is "dtype": "|S3" and I am back to my length problem: ['foo' 'bar'].

The solution with 'dtype': '|S6' is acceptable, but I need to encode my strings to bytes when indexing, which is annoying.

Start by running the reported Python reproduction and inspect the to_zarr, open_zarr, and append paths involved in dtype and string handling. Done means appending strings longer than the initial value preserves their complete contents while retaining the expected string behavior without requiring callers to encode values manually.