dandi / dandi/dandi-cli

Zarr depth limit rejects NWB chunk paths

Open
#1,919 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

I converted an IBL session while testing [hdmf-zarr's v3 migration](https://github.com/hdmf-dev/hdmf-zarr/pull/325) and tried uploading it to [sandbox dandiset 218696](https://sandbox.dandiarchive.org/dandiset/218696). DANDI 0.78.0 stopped at validation:

```text
[dandi_zarr.tree_depth_exceeded] Zarr directory tree more than 7 directories deep
```

The processed file has 34 files over the limit, including:

```text
processing/pose_estimation/RightCamera/PoseEstimationSeriesTubeTop/data/c/1/0
```

The NWB array path has five components. Zarr v3 adds `c/1/0` for the chunk coordinates. With the usual v2 encoding this would end in `data/1.0` and fit under the limit.

This numeric-only example reproduces it:

```python
import numpy as np
import zarr

group = zarr.open_group("deep.zarr", mode="w", zarr_format=3)
array = group.create_array("a/b/c/d/series/data", shape=(2, 2), chunks=(1, 1), dtype="float64")
array[:] = np.arange(4).reshape(2, 2)
```

Run `dandi validate deep.zarr` (outside a dandiset it also reports `DANDI.NO_DANDISET_FOUND`). The IBL upload also hit the string validation error reported in #1918. This example uses only numeric data to isolate the depth error.

I found the same problem in #983 from 2022. Satra pointed out that the nested NGFF chunk paths were legitimate. John asked whether to increase or remove the limit. Satra suggested seven to cover the use cases at the time, and Yarik agreed:

https://github.com/dandi/dandi-cli/issues/983#issuecomment-1108693135

The original proposal was concerned about accidentally uploading large directory trees as thousands of assets:

https://github.com/dandi/dandi-cli/issues/852#issuecomment-995976767

Do we still need this restriction inside a recognized Zarr asset? Could we remove it or exclude chunk coordinates from the count?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the numeric reproduction with `dandi validate deep.zarr` and trace the validation entry point that reports `dandi_zarr.tree_depth_exceeded`. Determine how recognized Zarr v3 chunk-coordinate directories are counted, then add coverage for the five-component array path; done means legitimate chunk paths no longer trigger the depth error while the intended directory-tree safeguard remains.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.