CONP-PCNO / CONP-PCNO/git-annex-remote-globus

URL support functionality notes

Open
#3 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Here is a protocol from running `git annex addurl` on `s3://` url which is handled via datalad special remote:

```
$> git annex initremote datalad type=external externaltype=datalad encryption=none
...
$> git annex addurl --pathdepth=-1 --debug s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt
...
[2019-11-08 10:50:13.671664748] git-annex-remote-datalad[1] <-- CLAIMURL s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt
[2019-11-08 10:50:13.671791652] git-annex-remote-datalad[1] --> DEBUG Encodings: filesystem utf-8, default utf-8
[2019-11-08 10:50:13.67187777] Encodings: filesystem utf-8, default utf-8
[2019-11-08 10:50:13.672111287] git-annex-remote-datalad[1] --> DEBUG Claiming url 's3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt'
[2019-11-08 10:50:13.672210539] Claiming url 's3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt'
[2019-11-08 10:50:13.672270751] git-annex-remote-datalad[1] --> CLAIMURL-SUCCESS
[2019-11-08 10:50:13.67239921] git-annex-remote-datalad[1] <-- CHECKURL s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt
[2019-11-08 10:50:14.040837149] git-annex-remote-datalad[1] --> CHECKURL-CONTENTS 4250 ds116/sub001/BOLD/task001_run001/QA/fd.txt
addurl s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt (from datalad) (to ds116/sub001/BOLD/task001_run001/QA/fd.txt) [2019-11-08 10:50:14.041350273] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2019-11-08 10:50:14.050977324] process done ExitSuccess
[2019-11-08 10:50:14.051141617] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
[2019-11-08 10:50:14.063291123] process done ExitSuccess
[2019-11-08 10:50:14.063796021] chat: git ["--git-dir=.git","--work-tree=.","check-ignore","-z","--stdin","--verbose","--non-matching"]

[2019-11-08 10:50:14.085449929] git-annex-remote-datalad[1] <-- TRANSFER RETRIEVE URL-s4250--s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt .git/annex/tmp/URL-s4250--s3&c%%openfmri%ds116%sub001%BOLD%task001_run001%QA%fd.txt
[2019-11-08 10:50:14.087122748] git-annex-remote-datalad[1] --> GETURLS URL-s4250--s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt http:
[2019-11-08 10:50:14.090879728] git-annex-remote-datalad[1] <-- VALUE
[2019-11-08 10:50:14.09178823] git-annex-remote-datalad[1] --> GETURLS URL-s4250--s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt https:
[2019-11-08 10:50:14.094266045] git-annex-remote-datalad[1] <-- VALUE
[2019-11-08 10:50:14.094895422] git-annex-remote-datalad[1] --> GETURLS URL-s4250--s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt s3:
[2019-11-08 10:50:14.098633146] git-annex-remote-datalad[1] <-- VALUE s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt
[2019-11-08 10:50:14.098871334] git-annex-remote-datalad[1] <-- VALUE
[INFO] Downloading 's3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt' into '.git/annex/tmp/URL-s4250--s3&c%%openfmri%ds116%sub001%BOLD%task001_run001%QA%fd.txt'
[2019-11-08 10:50:14.245314399] git-annex-remote-datalad[1] --> PROGRESS 4250
100% 4.15 KiB 26 KiB/s 0s[2019-11-08 10:50:14.246005387] git-annex-remote-datalad[1] --> TRANSFER-SUCCESS RETRIEVE URL-s4250--s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt
[2019-11-08 10:50:14.24666711] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
[INFO] Successfully downloaded s3://openfmri/ds116/sub001/BOLD/task001_run001/QA/fd.txt into .git/annex/tmp/URL-s4250--s3&c%%openfmri%ds116%sub001%BOLD%task001_run001%QA%fd.txt
ok
(recording state in git...)
...
```

So the workflow would be
- react with `CLAIMURL-SUCCES` for CLAIMURL those urls which start with `globus://[|]/` where `` (or ``) and `` are options of the special remote
- provide `CHECKURL-CONTENTS Size|UNKNOWN Filename` response for `CHECKURL` query by annex (for those matching URLs)
- on `TRANSFER RETRIEVE Key File` we could analyze the provided Key.
1. If Key is of URL- backend (starts with `URL-`) we could actually avoid using GETURLS (as we did in datalad) but just parse that key to extract the URL and corresponding path to be RETRIEVED
2. while transferring we need to provide back PROGRESS reports
3. the tricky part here is that for RETRIEVE we would like to support also regular/proper git annex special remote behavior, if data was stored in the layout of a regular special remote. So we might want first to check (yet to determine specifics) if a key is available as on regular annex special remote; and if not /no information -- run GETURLS, filter for the ones we care about and try to download using them instead
4. instead of i. + iii. -- may be we should just do GETURLS, regardless of the key, and only if that doesn't provide us any URLs we handle, then get to "regular special remote" way.

but overall summary -- we should be able to make it work as a proper git annex external special remote with GET/PUT and EXPORT while also supporting regular `annex addurl globus://...` functionality (thus `datalad addurs` could be used to establish "import" of already existing directories on globus; until git annex provides protocol/support for proper "import").

**BUT** I believe that git-annex might be the one which seems to "register url" (thus storing the ad-hoc globus:// url in git-annex branch in .web file for the key) for the key upon `addurl URL`. Ideally we should avoid that url being stored, but rather just store the path to the file (and version info) to that file assuming `globus://[|]/` prefix. that would allow for more flexible management (e.g. rename of the globus endpoint, or renaming/moving `fileprefix`), and minimize storage within git-annex branch. We might need to clarify that with @joeyh (Q: is it possible for special remote to announce that claimed url shouldn't be stored as a url for the file)

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named; start by tracing the external special-remote handlers for CLAIMURL, CHECKURL, TRANSFER RETRIEVE, GETURLS, and the proposed GET/PUT and EXPORT support. Resolve the open git-annex URL-storage question first; done means matching globus:// URLs can be claimed, checked, transferred with progress, and handled alongside regular special-remote behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.