Support NWB files stored as Zarr (`.nwb.zarr`)
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
Same context as the other two issues, [1](https://github.com/dandi/dandi-cli/issues/1918) and [2](https://github.com/dandi/dandi-cli/issues/1919) but now I have realized (I should have thought about this before) that dandi-cli has no support for NWB stored as Zarr: a `.nwb.zarr`. Currently, this is handled as a generic Zarr asset, so none of the NWB machinery ever sees it.
In my specific case described on the issues above two things happend:
* The validation does not stop the upload in its tracks if the referenced videos are missing.
* `dandi organize` skips the `.nwb.zarr` file silently, no entry, no warning, no error.
I am not sure adding support is mechanically hard but there is a dependency problem. Reading of the file is supported by the backend independent `pynwb.read_nwb`, which is able to handle the two backends as long as hdmf-zarr is installed. That covers `_get_pynwb_metadata`, and `validate` already goes through `pynwb.validate(paths=...)` on its main path. Then there are the following sites that read quickly and where doing the full nwb read is probably too heavy, they will require a backend specific branch:
* `get_nwb_version`, which opens the file only to read one root attribute.
* `get_object_id`, same thing for `object_id`.
* `get_neurodata_types` and its `_scan_neurodata_types` helper, which walk the raw tree looking for `neurodata_type`.
* `nwb_has_external_links`, the gate that refuses files whose content lives in other files. hdmf-zarr has links with a `source` too, so this ports the same way.
I can take the above if that makes sense for you but the thorny issue is dependencies. nwb.zarr will require Python 3.12 and zarr 3.2 or above, [see the version discussion in hdmf-zarr](https://github.com/hdmf-dev/hdmf-zarr/issues/371). Currently you have zarr pinned to `zarr>=2.18.0,<=3.1.5`. Is there a reason for this? How could we move forward?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the NWB handling around get_nwb_version, get_object_id, get_neurodata_types, _scan_neurodata_types, nwb_has_external_links, _get_pynwb_metadata, and validate. Review the zarr dependency constraint and the hdmf-zarr version discussion before deciding how the backend-specific branches should work. Done means .nwb.zarr files receive NWB metadata, validation, external-link checks, and organize handling rather than being treated as generic Zarr assets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100