validate should stop skipping/ignoring files
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
ATM validate considers only .nwb files so it is peacefully ignores other unknown extensions:
```shell
$> dandi validate /tmp/testds
Summary: No validation errors among 0 file(s)
2022-02-21 20:41:20,064 [ INFO] Logs saved in /home/yoh/.cache/dandi-cli/log/20220222014117Z-207803.log
$> ls -l /tmp/testds
total 16
lrwxrwxrwx 1 yoh yoh 108 Feb 18 15:22 1.unlocked -> .git/annex/objects/62/QQ/MD5E-s2--e2c942abe10bccd8b392aaf4d7f9832f/MD5E-s2--e2c942abe10bccd8b392aaf4d7f9832f
-rw------- 1 yoh yoh 20 Feb 18 14:56 123
lrwxrwxrwx 1 yoh yoh 108 Feb 18 14:50 124 -> .git/annex/objects/6G/wQ/MD5E-s4--df8b712c4fe20a0df933819665770165/MD5E-s4--df8b712c4fe20a0df933819665770165
-rw------- 1 yoh yoh 88 Feb 18 14:27 fs.log
```
`validate` operation should get refactored to support "multiple levels" of validation:
1. files validation: what we have now for .nwb files, but possibly add rudimentary support for others, e.g. video files (#918), basic .json/.yaml/.tsv format/consistency validation (that we can load them alright and there is no syntax etc errors)
2. dataset(s) validation:
- `dandiset.yaml` must be present at the path - it defines the top level of the dandiset
- need to validate `dandiset.yaml` to pass dandischema validation
- then two possible scenarios for "layout", depending on either we detect the dataset also be or contain BIDS dataset (has some `dataset_description.json` and it is a BIDS dataset (like https://gui.dandiarchive.org/#/dandiset/000108/draft/files?location=) or a "collection" of BIDS datasets and derivatives + rawdata (like https://gui.dandiarchive.org/#/dandiset/000026/draft/files?location=) )
- DANDI convention: all other assets (files) in that dandiset should follow our DANDI files naming convention (`sub-/sub-_ses-{session}[_entity-{value}]*_suffixes.{extensions}`)
- BIDS: we apply BIDS validator to those (sub)folders which have dataset_description.json (relevant work #896)
question in above "dataset(s) validation" on what to do to collections containing BIDS datasets, like https://gui.dandiarchive.org/#/dandiset/000026/draft/files?location= and which just follow BIDS recommendation (https://bids-specification.readthedocs.io/en/stable/02-common-principles.html#source-vs-raw-vs-derived-data) -- we might want to "DANDI-standardize" that convention as well to reduce arbitrary layouts (like https://github.com/dandisets/000052/tree/draft/dataset ). WDYT @satra on this and above?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.