dandi / dandi/dandi-cli

DANDI validation rejects Zarr v3 string arrays

Open
#1,918 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

I converted an IBL session while testing [hdmf-zarr's v3 migration](https://github.com/hdmf-dev/hdmf-zarr/pull/325) and tried uploading it to [sandbox dandiset 218696](https://sandbox.dandiarchive.org/dandiset/218696). `dandi upload` stopped at validation:

```text
[zarr.tensorstore_cannot_open] .../identifier
Error opening Zarr array with tensorstore
```

All 113 string arrays fail this check. The other 176 numeric and boolean arrays pass. Opening `identifier` directly with TensorStore gives:

```text
Error parsing object member "data_type": string data type is not one of the supported data types
```

This is enough to reproduce the validation error:

```python
import numpy as np
import zarr

group = zarr.open_group("strings.zarr", mode="w", zarr_format=3)
array = group.create_array("identifier", shape=(1,), dtype=np.dtypes.StringDType())
array[0] = "hello"
```

Then run `dandi validate strings.zarr` (outside a dandiset it also reports `DANDI.NO_DANDISET_FOUND`).

I used DANDI 0.78.0, TensorStore 0.1.85 and Zarr 3.3.0, overriding DANDI's `zarr<=3.1.5` pin. I also checked an array written with Zarr 3.1.5 and TensorStore rejects it with the same dtype error.

I think `_ts_validate_zarr3` is rejecting an encoding that Zarr-Python can read. Its docstring already says it should go away after the Zarr-Python 3 migration. Could we use Zarr-Python to validate these arrays? What do you think?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the `_ts_validate_zarr3` entry point and reproduce the failure with the provided Zarr v3 string-array example, then run `dandi validate strings.zarr`. Trace how the validation handles the reported string dtype and compare it with the arrays Zarr-Python can read. Done means string arrays validate successfully while the existing numeric and boolean array checks continue to pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
cli, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.