google / google/tensorstore

tensorstore cannot open vlen UTF8 string written with Zarr-Python

Open
#103 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.5k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

To reproduce:
```python
import zarr

zarr.save_array(
'/tmp/string.zarr',
np.array(['foo', 'bar'], dtype=object),
dtype=str,
)
tensorstore.open({'driver': 'zarr', 'kvstore': 'file:///tmp/string.zarr'}).result()
```
Raises:
```
ValueError: FAILED_PRECONDITION: Error opening "zarr" driver: Error reading local file "/tmp/string.zarr/.zarray": Error parsing object member "dtype": Unsupported zarr dtype: "|O" [tensorstore_spec='{\"context\":{\"cache_pool\":{},\"data_copy_concurrency\":{},\"file_io_concurrency\":{}},\"driver\":\"zarr\",\"kvstore\":{\"driver\":\"file\",\"path\":\"/tmp/string.zarr/\"}}']
```

Zarr-Python writes these arrays (see [documentation](https://zarr.readthedocs.io/en/stable/tutorial.html#string-arrays)) with an "object" dtype and `vlen-utf8` filter:
```
$ cat /tmp/string.zarr/.zarray
{
"chunks": [
2
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "|O",
"fill_value": 0,
"filters": [
{
"id": "vlen-utf8"
}
],
"order": "C",
"shape": [
2
],
"zarr_format": 2
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.