Derivatives can't leave the cluster that made them (STAMPED-D): per-derivative RIA with a machine-local URL
- Dominant language
- Python
- Stars
- 1
- Forks
- 4
- Avg merge
- 15h 39m
- Merged PRs (30d)
- 24
Description
A derivative that cannot leave the cluster that made it is not **D**istributable — and "publishable" implies collectable, so this gates M2. Today there is no supported way to get a produced derivative off the cluster; the 2026-07-19 Unity shakeout got its results out by hand, through four separate blockers.
## Observed
Jobs push content to each derivative's **own** output RIA (`/.babs/output_ria/`, one store per cell). `babs merge` + `datalad update` fast-forward only the working tree, so the campaign holds annex symlinks, not content. A clone elsewhere inherits the `output-storage` special-remote config whose URL is an absolute `ria+file:///scratch4/...` path on the cluster — nonexistent off it, so `datalad get` has nothing to fetch from, and git-annex does not transitively fetch. Cloning each derivative's RIA individually is exactly the per-derivative fiddling worth avoiding.
## Candidate fix — ORA over ssh
`output-storage` is a git-annex **ORA** special remote, and ORA speaks `ria+ssh://` as well as `ria+file://` (plain ssh, no daemon). Fetching straight out of the cluster's RIA without materializing anything in its working tree is what ORA is for. Registering a second access path to the *same* store is the idiomatic move:
```
git annex initremote --sameas=output-storage type=external externaltype=ora \
url=ria+ssh://:/.babs/output_ria
```
`--sameas` tells git-annex this is another route to a store whose keys it already knows, so nothing is re-registered and no disk is doubled. **Unverified** — confirm ORA's exact parameter spelling before relying on it. Still per-derivative, so it wants wrapping.
Alternatives priced and rejected for now: one campaign-level `create-sibling-ria --recursive` + recursive push (may have nothing to push, since content lives only in each `.babs/output_ria`); wholesale `rsync` of the campaign (carries all content, but every `ria+file://` URL then needs an `enableremote url=` fixup per derivative).
## What worked, as the interim
Reusable scripts, not mechababs code, until the produced shape settles. Four non-obvious steps:
1. **On the cluster, materialize content into the working trees first** — `datalad get /derivatives/`. Required: the remote end fetches from `origin` over ssh, not from the RIA, so anything still only in the RIA is unreachable. Costs a temporary duplication, reclaimable after.
2. **Clone with an absolute/ssh source.** A relative clone source breaks depth-2 submodule URL resolution (relative child URLs compound and resolve to a nonexistent path).
3. **`install(error): Target commit ... does not exist in the clone`** — study subdatasets are registered by their upstream URL, and upstream has never seen the commit registering *our* derivatives. Fix: `datalad -c 'datalad.get.subdataset-source-candidate-100={remoteurl-origin}/{path}' get ` — the leading `100` is a cost (beats the default 700), and the template composes at every nesting level. Half-installed study dirs must be removed first; datalad will not retry an install over an existing directory.
4. **`Remote origin does not have git-annex installed; setting annex-ignore`** — a non-interactive `ssh host 'cmd'` never runs the interactive preflight that puts git-annex on PATH, and datalad caches the conclusion **per dataset**, so it fails stickily and fixing PATH alone does nothing. Fix: make git-annex reachable non-interactively (`ssh 'which git-annex-shell'`), then `git -C config --unset remote.origin.annex-ignore` in every affected dataset.
Gotchas: globs cannot expand before subdatasets are installed; `/derivatives/*` also matches upstream's *published* derivatives; and never `datalad get -r` a derivative — its subdatasets are `containers/` (a ~15 GB SIF) and `sourcedata/raw/` (the whole raw dataset).
## Scope
Transport stays reusable scripts for now, deliberately: the machinery depends on decisions not yet made, and building it into the CLI would bake in a published shape we would then have to un-publish. It moves into mechababs once it clearly belongs there *and* the shape of what we produce is settled.
Related: #6 (abspaths / portability — the same root fact, that machine-local URLs get recorded where canonical ones belong).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by verifying git-annex ORA's exact initremote parameters and inspecting the existing output-storage configuration described in the issue. Define whether reusable scripts can provide per-derivative SSH access without duplicating content; implementation is not ready until the produced shape and transport decisions are settled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git
- Domain
- infrastructure, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100