DANDI validation rejects Zarr v3 string arrays
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
I converted an IBL session while testing [hdmf-zarr's v3 migration](https://github.com/hdmf-dev/hdmf-zarr/pull/325) and tried uploading it to [sandbox dandiset 218696](https://sandbox.dandiarchive.org/dandiset/218696). `dandi upload` stopped at validation:
```text
[zarr.tensorstore_cannot_open] .../identifier
Error opening Zarr array with tensorstore
```
All 113 string arrays fail this check. The other 176 numeric and boolean arrays pass. Opening `identifier` directly with TensorStore gives:
```text
Error parsing object member "data_type": string data type is not one of the supported data types
```
This is enough to reproduce the validation error:
```python
import numpy as np
import zarr
group = zarr.open_group("strings.zarr", mode="w", zarr_format=3)
array = group.create_array("identifier", shape=(1,), dtype=np.dtypes.StringDType())
array[0] = "hello"
```
Then run `dandi validate strings.zarr` (outside a dandiset it also reports `DANDI.NO_DANDISET_FOUND`).
I used DANDI 0.78.0, TensorStore 0.1.85 and Zarr 3.3.0, overriding DANDI's `zarr<=3.1.5` pin. I also checked an array written with Zarr 3.1.5 and TensorStore rejects it with the same dtype error.
I think `_ts_validate_zarr3` is rejecting an encoding that Zarr-Python can read. Its docstring already says it should go away after the Zarr-Python 3 migration. Could we use Zarr-Python to validate these arrays? What do you think?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the `_ts_validate_zarr3` entry point and reproduce the failure with the provided Zarr v3 string-array example, then run `dandi validate strings.zarr`. Trace how the validation handles the reported string dtype and compare it with the arrays Zarr-Python can read. Done means string arrays validate successfully while the existing numeric and boolean array checks continue to pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- cli, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100