dandi / dandi/dandi-utils

"Integration" with Open Science Data Federation

Open
#15 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
No language data
Stars
0
Forks
2
PR merge metrics
No merged PRs in 30d

Description

### Proposed change

yet to be figured out, but a user asked

Please clarify whether data saved in DANDI can be reached and used through OSDF.
OSDF: https://osg-htc.org/services/osdf

From https://osg-htc.org/docs/data/stashcache/overview/: The OSG operates the Open Science Data Federation (OSDF), which provides organizations with a method to distribute their data in a scalable manner to thousands of jobs without needing to pre-stage data at each site.

Some caching solutions are part of the framework: https://osg-htc.org/docs/data/stashcache/install-cache/

Overall -- it sounds like a solution allowing making (public or private ) local data resources available to remote compute resources through unified interface with some caching to make it efficient. So, if someone tackles it I guess OSDF DANDI users would benefit form

- uniform access to DANDI data via OSDF interfaces
- caching
- may be discovery of some kind of a resource

Wasn't sure where to seek clarification yet... or what it would entail for a DANDI resource to expose itself as part of OSDF. If there was some other example of a public resource exposed to OSDF, might have provided a template to follow. Searching github got to https://github.com/PerilousApricot/docs/blob/de67fc125efb334ca67b011bde179689f035ffda/docs/data/stashcache/vo-data.md?plain=1#L3 so I will email `help@opensciencegrid.org` for either such integration makes sense if there are examples

### Alternative options

regular means to access from S3 directly, via DANDI API, via datalad/git-annex, dandi-cli

### Who would use this feature?

Users of OSDF

### (Optional): Suggest a solution

yet to be figured out if there is a problem to be solved ;)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the linked OSDF and StashCache documentation and confirming with the referenced Open Science Grid contact whether DANDI data can be exposed through OSDF. Compare that possibility with the listed S3, DANDI API, datalad/git-annex, and dandi-cli access paths. Done requires a defined integration scope and documented feasibility or a clear explanation that no change is needed.

Written by the indexing model from the issue text.

Assessment

Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.