CONP-PCNO / CONP-PCNO/conp-dataset

Size or other cut-off points for building git-annex links

Open
#674 12 comments 0 reactions 4 assignees Claimed by @GHPBZ View on GitHub
Discussion Required Stale
Dominant language
Python
Stars
18
Forks
34
Avg merge
2d 1h
Merged PRs (30d)
4

Description

Right now, our general policy when ingesting new datasets is to build a git-annex link to every file in a dataset by default, with a couple of specific exceptions (README.md and DATS.json). However, the utility of building links rather than just storing small files directly in github is questionable, and in tests with the microstructure_informed_connectomics dataset, which contains ~11,300 files, building git-annex links to each file took nearly twice as long as building links only to files larger than a cut-off of 200kb (estimated by manual examination of some subdirectories) and downloading the rest directly.

Do we want to consider size-based or other criteria for which files get git-annex links (such as storing all text files directly) ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.