Publish from the retained SOCI OCI layout instead of re-downloading from the temp registry

Problem

The publish stage moves the full image over the network six times per SOCI-enabled target. Three of those are avoidable.

SociConvertWorkflow (posit_bakery/plugins/builtin/imagetools/soci.py:193-248) pulls every layer into a local OCI layout, converts it, pushes every layer back to the temp registry, then deletes the scratch directory in an unconditional finally:. OrasIndexCopyWorkflow then issues one oras cp per destination registry — and each one re-downloads from the temp registry, because ORAS keeps no content store between invocations.

The local layout holding every byte is discarded immediately before two copies re-download the same bytes.

Marginal cost per additional destination registry also halves: 2× → 1×.

SOCI is enabled on every image using this path — all three in images-connect, all three in images-workbench, and package-manager.

Validated

Against oras 1.3.3 with two local registries (one would dedup blobs and mask the effect):

• oras cp --from-oci-layout <layout>@<digest> <registry>/<repo>:t1,t2,t3 succeeds; all three tags resolve to the same digest. The multi-tag destination form OrasIndexCopyWorkflow builds (oras.py:268) works from a layout source.
• Counting blob GETs against the source registry, two sequential registry→registry copies fetched blobs both times (2 and 2). The 6× figure is measured, not assumed.

Risks to weigh

Peak disk under parallel stage 1 — the blocking one. _run_publish_stage1 is fanned out across a ParallelShellExecutor (imagetools.py:644), while stage 2 is deliberately kept ordered for Docker Hub tag display (commit 5c4882b3, #484, #759). Retained layouts must survive that parallel→ordered boundary, so all concurrent targets' layouts coexist instead of each being freed at the end of its own conversion.

Fusing stage 1 and stage 2 per target is not available — it would break the ordered push. Mitigations: delete src immediately after conversion (halves per-target peak, since src and out currently coexist), cap --jobs, or accept and document a ceiling.

Losing the temp -soci artifact removes a retry checkpoint. Today a failed destination copy retries straight from the temp registry. Without the push-back, failure after conversion means redoing pull and an expensive SOCI conversion.

A middle option worth considering first: keep the push-back and publish from the retained layout — 1 down + 1 up + 2 up = 4×. Still a third off today, preserves the checkpoint, changes less.

Push ordering. The layout push is by digest (f"{out_layout}@{digest}"); today's copies start from a tagged temp ref. Equivalence needs confirming against #484's reverse-order requirement.

Interaction with #755. Splitting jobs per OS and architecture increases target count, so more layouts coexist and the disk risk worsens — though each is smaller.

Scope

Publish leg only. Cache import, cache export, and PR builds are unaffected.

Related

• https://github.com/posit-dev/images-shared/issues/759
• https://github.com/posit-dev/images-shared/issues/591
• https://github.com/posit-dev/images-shared/issues/755
• https://github.com/posit-dev/images-shared/issues/484

Trace SociConvertWorkflow in posit_bakery/plugins/builtin/imagetools/soci.py:193-248, OrasIndexCopyWorkflow in oras.py:268, and _run_publish_stage1 in imagetools.py:644. First map the parallel stage-1 to ordered stage-2 lifecycle, then choose and validate a retained-layout strategy that accounts for disk usage, retry behavior, digest-based ordering, and the measured reduction in registry transfers.