Zarr depth limit rejects NWB chunk paths
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
I converted an IBL session while testing [hdmf-zarr's v3 migration](https://github.com/hdmf-dev/hdmf-zarr/pull/325) and tried uploading it to [sandbox dandiset 218696](https://sandbox.dandiarchive.org/dandiset/218696). DANDI 0.78.0 stopped at validation:
```text
[dandi_zarr.tree_depth_exceeded] Zarr directory tree more than 7 directories deep
```
The processed file has 34 files over the limit, including:
```text
processing/pose_estimation/RightCamera/PoseEstimationSeriesTubeTop/data/c/1/0
```
The NWB array path has five components. Zarr v3 adds `c/1/0` for the chunk coordinates. With the usual v2 encoding this would end in `data/1.0` and fit under the limit.
This numeric-only example reproduces it:
```python
import numpy as np
import zarr
group = zarr.open_group("deep.zarr", mode="w", zarr_format=3)
array = group.create_array("a/b/c/d/series/data", shape=(2, 2), chunks=(1, 1), dtype="float64")
array[:] = np.arange(4).reshape(2, 2)
```
Run `dandi validate deep.zarr` (outside a dandiset it also reports `DANDI.NO_DANDISET_FOUND`). The IBL upload also hit the string validation error reported in #1918. This example uses only numeric data to isolate the depth error.
I found the same problem in #983 from 2022. Satra pointed out that the nested NGFF chunk paths were legitimate. John asked whether to increase or remove the limit. Satra suggested seven to cover the use cases at the time, and Yarik agreed:
https://github.com/dandi/dandi-cli/issues/983#issuecomment-1108693135
The original proposal was concerned about accidentally uploading large directory trees as thousands of assets:
https://github.com/dandi/dandi-cli/issues/852#issuecomment-995976767
Do we still need this restriction inside a recognized Zarr asset? Could we remove it or exclude chunk coordinates from the count?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the numeric reproduction with `dandi validate deep.zarr` and trace the validation entry point that reports `dandi_zarr.tree_depth_exceeded`. Determine how recognized Zarr v3 chunk-coordinate directories are counted, then add coverage for the five-component array path; done means legitimate chunk paths no longer trigger the depth error while the intended directory-tree safeguard remains.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100