jaraco / jaraco/skeleton

Extending the reusable test-suite workflow in heavily-customized downstreams (e.g. setuptools)

Open
#211 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
150
Forks
38
Avg merge
1m
Merged PRs (30d)
1

Description

## Background

#199 factored the shared `test`/`collateral` setup into a reusable `test-suite.yml` (reusable workflows having been chosen over composite actions per the discussion in #162 / #163). For most consumers this is a clear win: `main.yml` shrinks to thin `uses:` calls, and the shared steps + env live in one place.

For heavily-customized downstreams it's a harder fit. setuptools is the motivating case. Its `test` job interleaves a lot of project-specific work *with* the shared steps:

- pre-building the sdist/wheel and exporting `PRE_BUILT_SETUPTOOLS_*`
- a cached download of setupcfg examples
- `SETUPTOOLS_USE_DISTUTILS` and a pypy `continue-on-error`
- publishing coverage

Plus two jobs that can't use the reusable workflow at all: `test_cygwin` (every step runs under a Cygwin bash shell) and `integration` (separate, tag-gated).

## The friction

1. **You can't inject steps into a reusable workflow.** A calling job may only set `uses`/`with`/`secrets`/`needs`/`if`/`strategy`/`permissions` — never `steps`. So setuptools can't "call the shared workflow and add its pre-build steps"; the custom steps have nowhere to live in the caller.

2. **So the only way to keep the pattern is to fork `test-suite.yml`** and add the project steps there, gated by an input (setuptools uses a `full-test` flag). But that customizes *the skeleton's own shared file* — the one the skeleton will keep iterating on — so every future skeleton update to `test-suite.yml` now conflicts. It inverts the usual "customize `main.yml`, inherit `test-suite.yml`" contract.

3. **Env can't be shared with decoupled jobs.** A called reusable workflow doesn't inherit the caller's top-level `env:`, and `test_cygwin`/`integration` can't call it, so the shared env block (`FORCE_COLOR`, `PIP_*`, `TOX_OVERRIDE`) ends up duplicated across `test-suite.yml` and each decoupled job.

setuptools' current resolution (in the pending skeleton merge) is Option-D-style: a forked `test-suite.yml` with a `full-test` input gating the setuptools-specific steps, and `test_cygwin`/`integration` decoupled with duplicated env. It works, but it accepts (2) and (3).

## Question

Could the skeleton offer extension points so a downstream can customize without forking `test-suite.yml`? A couple of directions:

- **Convention hooks** — `test-suite.yml` runs a downstream-provided script at a well-known path if it exists (e.g. a pre-tox `./.github/pre-test` and/or a post-tox hook). Degenerate/no-op by default; setuptools would move its pre-build / download-cache logic into such a script, keeping `test-suite.yml` upstream-pristine.
- **Composite actions for the genuinely shared concerns** (setup-python + tox; publish-coverage) that *both* `test-suite.yml` and decoupled jobs like `test_cygwin` could call — which would also address the env/coverage duplication in (3). (Acknowledging the composite-action tradeoffs already raised in #162 — collapsed/ugly log output in particular.)

Curious whether hooks, composites, or "downstreams just fork `test-suite.yml` and own it" is the intended story for cases like setuptools.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the reusable test-suite.yml and its main.yml callers, then compare the setuptools jobs described in the issue, including test_cygwin and integration. The work is done when the project agrees on an extension strategy—hooks, composite actions, or downstream-owned forks—and documents how customized consumers should use it without unresolved ambiguity.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, yaml
Domain
build-system, ci-cd
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.