Publishing design: how a produced derivative becomes a published one

A handful of separately-surfaced questions turn out to be one thing: **what happens when a derivative we produced becomes a published one.** Collected here so they stay together; each can be promoted out — or punted out of this milestone — as it sharpens.

**Why this is M2 and not a later concern:** a derivative registered with a local `./derivatives/` URL is not correctly recorded. If fixing that turns out to require anything at *data production* time — a different URL written at `babs init`, a different annex backend, a pointer pre-registered before the object exists — then a dataset produced today has to be redone. That is the M2 litmus. Whichever parts turn out to be pure post-processing can leave M2 once we know that.

## The goal this serves

*All our pointers upstream, our bits local, and the bits provably identical.*

Every object separates **where to get it** (submodule URL, annex remote URL — should be canonical/upstream) from **what it is** (gitlink sha, annex key = content hash — the proof). Because keys are recorded in the git tree and gitlinks pin child shas, **one commit sha is the root of trust for the whole nest**: local bits verify against the name, never by diffing against upstream. That is how STAMPED's self-contained pull (bits local) and its portable/distributable pull (resolvable pointers) stop being a tradeoff — content-addressing buys both.

## Settled

**Publishing after the fact does not weaken provenance.** Downstream stages chain off an upstream derivative that is still unpublished when they consume it. Identity and location are different claims: what a run consumed is fixed at run time by content (annex key, dataset id, commit) and never changes; where that object can *now* be fetched is additive location information. Adding a location is not rewriting history — git-annex separates location tracking from content identity for exactly this reason, and a `.gitmodules` URL is a location statement while the unchanged gitlink sha is the identity statement. The impure move would be changing *which* object is claimed, which this is not.

Consequence: **review-before-publish and don't-block-processing are not in conflict.** Worth confirming against BEP028's view, since it is a provenance-model claim.

## Open questions

1. **Local URLs in `.gitmodules`.** Our derivatives are registered into the study with `./derivatives/` paths, sitting beside upstream's published derivatives which carry real OpenNeuroDerivatives URLs — that difference *is* the not-yet-published tell. Publishing must end with the public URL recorded. Rewrite in place, or re-clone from the public URL? And can it be done entirely after production, or does something have to change at init?
2. **When does a pointer become upstream?** A pointer cannot be upstream before the object exists upstream. Rewrite at publish time, or pre-register the eventual URL at creation? (A sharp edge already: a chained input's `origin_url` cannot be recorded before its upstream is produced.)
3. **Multi-location keys.** "We hold a copy so we don't depend on upstream" really means *one key, several registered locations, one canonical URL* — git-annex location tracking / ORA `--sameas`. That reframes transport (#79) from "how do I move bits" to "how does a key learn it has a second home."
4. **Annex backend.** "Provably identical" is only as strong as the key. If derivative content is `MD5E` (OpenNeuro's convention) rather than `SHA256E`, that is corruption detection, not integrity. Either move, or accept `MD5E` knowingly — but this is a production-time choice, so it is the item most likely to force a redo. Note zipping is temporary: once outputs land unzipped this is ~700 keys per subject rather than one, changing both the stakes and the cost of switching.
5. **`--reference-if-able` as an accelerator.** `datalad clone "$public_url" new --reference-if-able /path/to/local --dissociate` clones from the canonical URL (so the *recorded* URL is the published one) while borrowing git objects locally so nothing is re-transferred, then dissociates to leave a standalone repo. `datalad clone` does pass git clone options through, with datalad's own caveat that not all options behave as expected, so it needs a real trial. Caveats: it shares **git objects only** (annex content still needs its own route); it does not fix `.gitmodules` by itself; and its value is contingent on the publish flow being re-clone-based. A technique to reach for once publishing is designed — not a candidate design.

## The unifying observation

Most of this is one fact wearing different hats: **machine-local URLs get recorded where canonical ones belong.** The local `./derivatives/...` entries, the `ria+file://` transport dead-end (#79), and the per-run chained-input `origin_url` injection are all that same fact.

Related: #79 (getting content off the cluster — concrete, separately tracked), #6 (abspaths / rerun portability).

No files or tests are named. Start by resolving the open questions in this issue, checking the settled provenance claim against BEP028, and reading related issues #79 and #6. Done means a decided publishing design that addresses canonical URLs, pointer locations, multi-location annex keys, and the production-time backend choice.

Written by the indexing model from the issue text.