dandi / dandi/dandi-schema

Proposal: Add Assets Directory to Dandisets for easier exploration

Open
#332 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
7
Forks
15
PR merge metrics
No merged PRs in 30d

Description

TLDR:
Allow users to upload a table that maps relevant experimental conditions to specific assets for quick exploration and systematic description.

## The problem
Neurophysiology datasets are highly heterogeneous, spanning multiple modalities, experimental conditions, pharmacological interventions, cell types, genotypes, etc. This diversity often results in a single study or dataset containing many different kinds of data and conditions, making it difficult for users to navigate without specialized knowledge.

For example I have been working with [Dandiset 001538](https://dandiarchive.org/dandiset/001538) which includes intracellular current and voltage clamp, two-photon series imaging, image stacks for spine density, line scans to measure dendritic excitability, behavioral videos, behavioral assessments, and acetylcholine sensor imaging. It further spans conditions such as different pharmacological manipulations, multiple genotypes, and diverse cell types, all distributed across different NWB files:

* **Experimental Measurements**: AChFP, AIMs, ConfSpine, DendExc, SomExc, SpineDens, oEPSC, video
* **Cell Types**: dSPN, iSPN, pan
* **Experimental States**: CTRL, LesionedControl, OFF, ON, OffState, OnState, PD
* **Pharmacology**: D1RaSch, D2RaSul, M1RaOxoM, M1RaThp, M1RaTri, none
* **Genotypes**: CDGIKO, M1RCRISPR, WT, iSPN-M1RKO, iSPNM1RKO

Currently, there is no standardized way to annotate dandisets so that a user can quickly determine which NWB files correspond to which modalities or conditions. The current workflow requires users to download or remotely open NWB files and inspect their internal structure to identify relevant data. This is problematic because it requires NWB-specific expertise, consumes unnecessary bandwidth for the archive, and slows down exploratory use of dandisets.

## Proposed Solution

Introduce a **user-provided table** within each dandiset that maps experimental conditions, modalities, and assets. Example:

| experimental_condition | measurement_type | genotype | asset_path |
| ---------------------- | ------------------------- | ----------- | ------------------------------------------------------------------------ |
| ON | dendritic_excitability | WT_variant | asset_path_1 |
| OFF | somatic_excitability | KO_line | asset_path_2 |
| CTRL | acetylcholine_measurement | CRISPR_mut | asset_path_3 |
| PD | opto_genetic_stimulation | dSPN_type | asset_path_4 |

The table should contain **fixed references as asset identifiers** (here I am using asset_path) for unambiguous mapping. This will allow users to fetch the table, filter it by condition of interest and then only download or stream those assets.

A minimal version would simply require the uploader to provide and maintain a table with mappings. An enhanced version could allow local path specification during upload, automatically converting them into asset references. Future extensions might include validation tools, web-based editors, or integration with existing metadata standards but let's discuss that if we get there.

### Related considerations
Restricting dandisets to one modality is unrealistic. Publications often include multimodal data and more than one experimental setup. Users typically create a dandiset publication per paper, and forcing them to split across multiple dandisets would add unnecessary hurdles.

Another option is to encode this information into the `session_id`. This approach feels natural to users, and currently has the advantage that `session_id` gets translated into the asset path, avoiding the need to fetch the NWB file. I adopted a version of this for the dandiset above, but at a general level it is problematic: it overloads the purpose of the session ID with a mini embedded language which does not follow separation of concerns. Plus, parsing a per-case language is brittle, and it risks clashing with asset naming conventions.

We could adopt or modify something like BIDS [sessions.tsv](https://bids-specification.readthedocs.io/en/v1.6.0/06-longitudinal-and-multi-site-studies.html#sessions-file) but at the moment that is subject scope which is not as general as needed here.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.