con / con/mechababs

Derivatives can't leave the cluster that made them (STAMPED-D): per-derivative RIA with a machine-local URL

Open
#79 0 comments 0 reactions 0 assignees View on GitHub
automation provenance
Dominant language
Python
Stars
1
Forks
4
Avg merge
15h 39m
Merged PRs (30d)
24

Description

A derivative that cannot leave the cluster that made it is not **D**istributable — and "publishable" implies collectable, so this gates M2. Today there is no supported way to get a produced derivative off the cluster; the 2026-07-19 Unity shakeout got its results out by hand, through four separate blockers.

## Observed

Jobs push content to each derivative's **own** output RIA (`/.babs/output_ria/`, one store per cell). `babs merge` + `datalad update` fast-forward only the working tree, so the campaign holds annex symlinks, not content. A clone elsewhere inherits the `output-storage` special-remote config whose URL is an absolute `ria+file:///scratch4/...` path on the cluster — nonexistent off it, so `datalad get` has nothing to fetch from, and git-annex does not transitively fetch. Cloning each derivative's RIA individually is exactly the per-derivative fiddling worth avoiding.

## Candidate fix — ORA over ssh

`output-storage` is a git-annex **ORA** special remote, and ORA speaks `ria+ssh://` as well as `ria+file://` (plain ssh, no daemon). Fetching straight out of the cluster's RIA without materializing anything in its working tree is what ORA is for. Registering a second access path to the *same* store is the idiomatic move:

```
git annex initremote --sameas=output-storage type=external externaltype=ora \
url=ria+ssh://:/.babs/output_ria
```

`--sameas` tells git-annex this is another route to a store whose keys it already knows, so nothing is re-registered and no disk is doubled. **Unverified** — confirm ORA's exact parameter spelling before relying on it. Still per-derivative, so it wants wrapping.

Alternatives priced and rejected for now: one campaign-level `create-sibling-ria --recursive` + recursive push (may have nothing to push, since content lives only in each `.babs/output_ria`); wholesale `rsync` of the campaign (carries all content, but every `ria+file://` URL then needs an `enableremote url=` fixup per derivative).

## What worked, as the interim

Reusable scripts, not mechababs code, until the produced shape settles. Four non-obvious steps:

1. **On the cluster, materialize content into the working trees first** — `datalad get /derivatives/`. Required: the remote end fetches from `origin` over ssh, not from the RIA, so anything still only in the RIA is unreachable. Costs a temporary duplication, reclaimable after.
2. **Clone with an absolute/ssh source.** A relative clone source breaks depth-2 submodule URL resolution (relative child URLs compound and resolve to a nonexistent path).
3. **`install(error): Target commit ... does not exist in the clone`** — study subdatasets are registered by their upstream URL, and upstream has never seen the commit registering *our* derivatives. Fix: `datalad -c 'datalad.get.subdataset-source-candidate-100={remoteurl-origin}/{path}' get ` — the leading `100` is a cost (beats the default 700), and the template composes at every nesting level. Half-installed study dirs must be removed first; datalad will not retry an install over an existing directory.
4. **`Remote origin does not have git-annex installed; setting annex-ignore`** — a non-interactive `ssh host 'cmd'` never runs the interactive preflight that puts git-annex on PATH, and datalad caches the conclusion **per dataset**, so it fails stickily and fixing PATH alone does nothing. Fix: make git-annex reachable non-interactively (`ssh 'which git-annex-shell'`), then `git -C config --unset remote.origin.annex-ignore` in every affected dataset.

Gotchas: globs cannot expand before subdatasets are installed; `/derivatives/*` also matches upstream's *published* derivatives; and never `datalad get -r` a derivative — its subdatasets are `containers/` (a ~15 GB SIF) and `sourcedata/raw/` (the whole raw dataset).

## Scope

Transport stays reusable scripts for now, deliberately: the machinery depends on decisions not yet made, and building it into the CLI would bake in a published shape we would then have to un-publish. It moves into mechababs once it clearly belongs there *and* the shape of what we produce is settled.

Related: #6 (abspaths / portability — the same root fact, that machine-local URLs get recorded where canonical ones belong).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by verifying git-annex ORA's exact initremote parameters and inspecting the existing output-storage configuration described in the issue. Define whether reusable scripts can provide per-derivative SSH access without duplicating content; implementation is not ready until the produced shape and transport decisions are settled.

Written by the indexing model from the issue text.

Assessment

Tech stack
git
Domain
infrastructure, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.