CONP-PCNO / CONP-PCNO/git-annex-remote-globus
Joey-on-the-change-of-FRDR-end-point
- Dominant language
- Python
- Stars
- 3
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
> I would like to ask you a question about git annex way to manage unused data. I
> built a git annex special remote using Globus.org as a datastore and I am
> setting up a new datalad dataset retrieving files content from Globus.
>
> So I set the files keys present with 1 and register the related urls as
> expected. Now, the input parameter (prefix) to initremote was wrong, hence my
> registered urls were all wrong, so I run 'git annex enableremote [new_prefix]'.
Did you use "git-annex registerurl" for that, or did the external
special remote use the SETURLPRESENT message?
The only way that initremote/enableremote would affect the url is if a
special remote does use SETURLPRESENT.
If the special remote does use SETURLPRESENT, there is also a SETURLMISSING
that it could use to remove the urls you want to remove. But, that might
not be the best fit for your situation, I think.
There is the `git-annex rmurl` command, that can remove an url from an
annexed file. Probably what you want, I think.
> This is all fine, git annex updates the prefix and builds new, now correct,
> urls. As you can imagine at this point, git annex has 2 urls per file (one
> wrong and one correct) and does not know what to do and fails.
Hmm, normally when git-annex has more than one url for a single file, it
will try all the urls, and only fails if all of the urls are not
accessible. So, I'm curious what this failure looks like.
> Admittedly it would be useful to have a command that clears content of internal
> git annex files (in the git annex branch) and provides a clean uninitialized
> repository without needing to set to 0 or 1 the whole previous history. For
> example, I have the case that the datasets in globus move to different
> locations and git annex enableremote must be re-run with the new updated
> parameters, and everything should work as expected. So I am thinking of
> something like 'git annex enablremote [new_parameter] --override'
Hmm, what data in the git-annex branch would that remove, would it be
all the data about that remote? Including removing information about
what keys are stored in the remote?
There is a way to do that, it's just to run `git-annex dead` with the
name or uuid of the remote, and then `git-annex initremote` a new
remote name.
But if you don't want to remove the information about what keys are
stored in the remote, using `git-annex rmurl` seems like a better
approach.
Would it help to be able to run a pipeline like this to find and delete
them all?
git-annex whereis --format '${file}\0${url}' | grep globus://wrong_prefix/ | git-annex rmurl --batch -z
(Not yet implemented but something like that could be added.)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the issue's proposed override workflow with the mentioned git-annex commands: rmurl, whereis, dead, initremote, and enableremote. Decide whether the requested behavior should remove URL records or all metadata for the remote; done means the intended cleanup or reconfiguration is unambiguous and works for the wrong Globus prefix case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100