Add ability to create subdatasets per path/glob
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 5
Description
Generated with the help of Claude Opus 4.7
## Summary
Extend `backups2datalad` to mint DataLad subdatasets for configurable path globs/regexes within a Dandiset, analogous to how `.zarr` directories are already handled as submodules.
## Motivation
This came out of a [discussion](https://github.com/dandi-compute/code/issues/87#issuecomment-4499305603) about how to organize derivatives and other large/modular components of a Dandiset without:
- Inflating inode usage in working trees (currently a full clone of e.g. `dandisets/000026` requires ~28,620 inodes even without zarr submodules, and annexed files require ~3x inodes each — easily hitting user inode quotas).
- Forcing users to clone entire `git-annex` tree just to work with a subset (sparse checkout can save local worktree).
The `.zarr` submodule pattern already demonstrates that fine-grained modular composition works well with DataLad. Generalizing it to arbitrary path patterns would let us apply the same benefits to derivatives, large NWB collections, or any other logically-separable subset.
## Proposed behavior
- Allow specifying one or more path globs/regexes (per-Dandiset, or via a default config) that should be split out as DataLad subdatasets.
- On backup, paths matching those patterns get their own subdataset (registered as a submodule in the parent), similar to the current `.zarr` handling.
- Subdatasets should be independently cloneable so that users can pull only the modules they need.
## Targets for Application
This would apply mainly to any BIDS derivative which has dandisets as an outer key
Note there is nothing preventing some non-DANDI maintainer from doing the same thing for some meta-analysis re-use as a BIDS-Study or derivative; surely we thus especially do not wish to encourage the 'duplicate the Dandisets' argument for a scaling of (number of Dandisets $\times$ number of reusers)
I suppose it could also WLOG apply to ANY/ALL outer subdirectories of monster Dandisets such as 000026 and could resolve some of the challenges faced over there
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.