dandi / dandi/dandi-cli

Support NWB files stored as Zarr (`.nwb.zarr`)

Open
#1,920 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

Same context as the other two issues, [1](https://github.com/dandi/dandi-cli/issues/1918) and [2](https://github.com/dandi/dandi-cli/issues/1919) but now I have realized (I should have thought about this before) that dandi-cli has no support for NWB stored as Zarr: a `.nwb.zarr`. Currently, this is handled as a generic Zarr asset, so none of the NWB machinery ever sees it.

In my specific case described on the issues above two things happend:
* The validation does not stop the upload in its tracks if the referenced videos are missing.
* `dandi organize` skips the `.nwb.zarr` file silently, no entry, no warning, no error.

I am not sure adding support is mechanically hard but there is a dependency problem. Reading of the file is supported by the backend independent `pynwb.read_nwb`, which is able to handle the two backends as long as hdmf-zarr is installed. That covers `_get_pynwb_metadata`, and `validate` already goes through `pynwb.validate(paths=...)` on its main path. Then there are the following sites that read quickly and where doing the full nwb read is probably too heavy, they will require a backend specific branch:

* `get_nwb_version`, which opens the file only to read one root attribute.
* `get_object_id`, same thing for `object_id`.
* `get_neurodata_types` and its `_scan_neurodata_types` helper, which walk the raw tree looking for `neurodata_type`.
* `nwb_has_external_links`, the gate that refuses files whose content lives in other files. hdmf-zarr has links with a `source` too, so this ports the same way.

I can take the above if that makes sense for you but the thorny issue is dependencies. nwb.zarr will require Python 3.12 and zarr 3.2 or above, [see the version discussion in hdmf-zarr](https://github.com/hdmf-dev/hdmf-zarr/issues/371). Currently you have zarr pinned to `zarr>=2.18.0,<=3.1.5`. Is there a reason for this? How could we move forward?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the NWB handling around get_nwb_version, get_object_id, get_neurodata_types, _scan_neurodata_types, nwb_has_external_links, _get_pynwb_metadata, and validate. Review the zarr dependency constraint and the hdmf-zarr version discussion before deciding how the backend-specific branches should work. Done means .nwb.zarr files receive NWB metadata, validation, external-link checks, and organize handling rather than being treated as generic Zarr assets.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.