dandi / dandi/backups2datalad

Add ability to create subdatasets per path/glob

Open
#99 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
1
Avg merge
2d 16h
Merged PRs (30d)
5

Description

Generated with the help of Claude Opus 4.7

## Summary

Extend `backups2datalad` to mint DataLad subdatasets for configurable path globs/regexes within a Dandiset, analogous to how `.zarr` directories are already handled as submodules.

## Motivation

This came out of a [discussion](https://github.com/dandi-compute/code/issues/87#issuecomment-4499305603) about how to organize derivatives and other large/modular components of a Dandiset without:

- Inflating inode usage in working trees (currently a full clone of e.g. `dandisets/000026` requires ~28,620 inodes even without zarr submodules, and annexed files require ~3x inodes each — easily hitting user inode quotas).
- Forcing users to clone entire `git-annex` tree just to work with a subset (sparse checkout can save local worktree).

The `.zarr` submodule pattern already demonstrates that fine-grained modular composition works well with DataLad. Generalizing it to arbitrary path patterns would let us apply the same benefits to derivatives, large NWB collections, or any other logically-separable subset.

## Proposed behavior

- Allow specifying one or more path globs/regexes (per-Dandiset, or via a default config) that should be split out as DataLad subdatasets.
- On backup, paths matching those patterns get their own subdataset (registered as a submodule in the parent), similar to the current `.zarr` handling.
- Subdatasets should be independently cloneable so that users can pull only the modules they need.

## Targets for Application

This would apply mainly to any BIDS derivative which has dandisets as an outer key

Note there is nothing preventing some non-DANDI maintainer from doing the same thing for some meta-analysis re-use as a BIDS-Study or derivative; surely we thus especially do not wish to encourage the 'duplicate the Dandisets' argument for a scaling of (number of Dandisets $\times$ number of reusers)

I suppose it could also WLOG apply to ANY/ALL outer subdirectories of monster Dandisets such as 000026 and could resolve some of the challenges faced over there

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.