Automattic / Automattic/wp-codebox

Publish pinned release artifacts: consumers fall back to building @main every CI run, so every upstream regression breaks all downstream CI live

Open
#1,703 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16
Forks
4
Avg merge
59m
Merged PRs (30d)
131

Description

## Summary

When the WP Codebox CLI release artifact is **not published**, downstream CI setups fall back to a **source install of `Automattic/wp-codebox@main` on every run**. That means every consumer's CI silently tracks tip-of-main, so **any regression merged to `wp-codebox@main` breaks all downstream CI immediately and simultaneously** — with no version pinning to insulate them. This turns wp-codebox `main` into an un-pinnable, always-latest dependency for every consumer's test gate.

## Concrete incident (Exhibit A)

On 2026-07-03, a mount-path change (PR #1698 "canonical-input-mount-paths", commit `242e36f8`) relocated the plugin-under-test out of the WordPress plugins tree, breaking composer autoloading of plugin classes inside the Playground runtime. Downstream, **every** data-machine PR's Test job went red with:

```
wordpress.phpunit could not run: Error: Class "DataMachine\Core\OAuth\BaseAuthProvider" not found
at /tmp/wp-codebox-inputs/0-data-machine-.../tests/Unit/Abilities/AuthAbilitiesTest.php:17
```

Timeline:
- **2026-07-02**: data-machine Test passed — 1370 tests green (built from `@main` source-install fallback).
- **2026-07-03 ~00:34 UTC**: #1698 merged to wp-codebox main.
- **2026-07-03**: identical install path, Test now crashes at collection. The **only** variable that changed was `wp-codebox@main`.

Root cause was fixed in #1699 (merged) — but the point of this issue is the **delivery mechanism that let a single upstream merge break all consumers at once**, not that specific regression.

## Why it happens

CI setup logs show the artifact-missing fallback explicitly:

```
Installing WP Codebox CLI from release artifact (.../wp-codebox-cli-linux-x64.tar.gz)...
WP Codebox release artifact not published at ...; falling back to source install
Installing WP Codebox CLI (https://github.com/Automattic/wp-codebox.git@main)...
```

The install ref defaults to `main` (`HOMEBOY_WP_CODEBOX_REF:-main` in the homeboy-extensions setup path), and consumers frequently **cannot override it** (e.g. a reusable GitHub workflow invoked via `uses:` does not inherit the caller's `env:`, and no codebox-ref input is exposed). So consumers are structurally stuck on `@main` with no pin lever.

## Requested fix (any of these breaks the "everyone tracks main" coupling)

1. **Publish versioned release artifacts** for each tagged release (e.g. `wp-codebox-cli-linux-x64.tar.gz` per `vX.Y.Z`) so the primary install path is a pinned artifact, not a source build of tip-of-main.
2. **Default the source-install fallback to the latest published release tag** rather than `main`, so a missing artifact doesn't silently escalate to bleeding-edge.
3. **Expose a first-class ref/version input** on the setup path so consumers can pin (and reusable-workflow consumers can pass it through), rather than relying on a caller `env:` that reusable workflows drop.

## Impact if unaddressed

Every future regression merged to `wp-codebox@main` will again break all downstream CI at once, with no per-consumer insulation — forcing each consumer into ad-hoc stopgaps (e.g. dropping the affected CI leg) instead of pinning to a known-good release. This is a systemic supply-chain fragility, not a one-off.

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace the homeboy-extensions setup path described in the issue, including the HOMEBOY_WP_CODEBOX_REF:-main fallback and reusable GitHub workflow inputs. Check how release artifacts are selected and published, then define done as consumers using a versioned artifact or explicitly passable ref instead of silently installing @main.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, devops, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.