con / con/nwb2bids

Add documentation on how to produce DataLad BIDS dataset: 2 possible setups

Open
#54 3 comments 0 reactions 1 assignee Claimed by @yarikoptic View on GitHub
documentation
Dominant language
Python
Stars
5
Forks
6
Avg merge
4h 6m
Merged PRs (30d)
1

Description

### approach 1: sourcedata/

Following the YODA style of containment and actual BIDS layout recommendations we could produce a BIDS dataset with

- sourcedata/
- `dandiset/` -- git submodule pointing to the dandiset from https://github.com/dandisets

then as a result of applying `nwb2bids` we could have multiple `--files-mode` (see dandi-cli which has/implemented them):
- `symlink` : would have `.nwb` files to become symlinks to original location, and if from under `sourcedata/` -- pointing to under `sourcedata/dandiset/...`
- (later) add a mode for `annex-copy` which would just copy the annex-key and establish the same annex key as original `sourcedata/dandiset/` and copy availability information (as we already have that file on s3)

Then if you use `datalad-fuse` on top I would expect it all work since datalad-fuse would provide "real file" view for the `sourcedata/dandiset/` dataset.

### approach 2: 3-branch method

A somewhat "slicker" alternative workflow could be similar to the one I demonstrated generated by datalad for "good old" openfmri datasets by our elderly https://github.com/datalad/datalad-crawler/blob/master/datalad_crawler/pipelines/simple_with_archives.py pipeline which produces 3 branches:
- `incoming` -- that is where it is the "original" content (here it would be current `draft` branch)
- `incoming-processed` -- that is where content of `incoming` for a new commit is processed into establishing a merge (it is not really a "merge" but rather: clean current worktree, take that tree of `incoming` and process (nwb2bids), and commit the result. (could be `draft-nwb2bids`)
- `master` -- true merges of the `incoming-processed` into `master` which could have its own content (e.g. `phenotype/` or `stimuli/` or 'code/') or changes (e.g. extended `dataset_description.json`) . (here could be just `bids` branch)

This way then it becomes feasible to inspect diffs in any of the branches or between them easily while git-annex would take care about efficient manipulation of content. It is quite amazing to be able to `git checkout bids` or `git checkout -` back to `draft` in a second or two.

edit: here is how it looks on https://datasets.datalad.org/openfmri/ds000001/.git repo in gitk:

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.