typelevel / typelevel/sbt-typelevel

Allow raw GHA expressions in matrix axes (and outputs)

Open
#887 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
185
Forks
62
PR merge metrics
No merged PRs in 30d

Description

Problem

WorkflowJob.matrixAdds: Map[String, List[String]] runs every value through the renderer's wrap helper, which YAML-quotes strings that look syntactically suspicious. There's no way to tell the renderer "this is a raw GitHub Actions expression — emit it unquoted."

This makes the runtime-discovered matrix pattern impossible to express through the API. Concretely, what GitHub Actions wants is:

strategy:
  matrix:
    test: ${{ fromJSON(needs.discover.outputs.tests) }}

…where test: is set to the expression itself, which evaluates to an array. There's no list literal — the whole axis is the expression.

Through matrixAdds the closest you can get is Map("test" -> List("${{ fromJSON(...) }}")), which renders as:

test: ['${{ fromJSON(needs.discover.outputs.tests) }}']

— a literal one-element list whose single string happens to contain an expression. GHA evaluates the expression but coerces the resulting array to a string and uses it as one matrix value, which is not what we want.

The same shape problem appears in WorkflowJob.outputs: Map[String, String]: the renderer auto-wraps each value in ${{ … }}, so passing "${{ steps.x.outputs.y }}" produces ${{ ${{ steps.x.outputs.y }} }}. Here it's papered over by documentation ("don't include the wrapping"), but it's the same root cause: the type doesn't distinguish "literal" from "expression."

Worked-around example

In polyvariant/smithy-trait-codegen-scala#39, we wanted exactly the runtime-discovered matrix shape (a discover job lists scripted-test folders, the scripted job fans out across them × the Scala/sbt versions). Because the matrixAdds API can't produce raw expressions, we ended up with:

  • a placeholder string (PATCH_scripted_tests_from_needs) passed through matrixAdds,
  • a custom githubWorkflowGenerateWithMatrixPatch task that runs githubWorkflowGenerate and then string-replaces the placeholder on disk with the ${{ fromJSON(...) }} expression,
  • a custom githubWorkflowCheckWithMatrixPatch task that snapshots the on-disk file, regenerates, re-applies the patch in memory, compares, and restores — to keep githubWorkflowCheck-style staleness detection working,
  • a third string-replace inside the patch to reroute the workflow's own "Check that workflows are up to date" step from sbt githubWorkflowCheck to the patched version, so both sides see the same form.

That's ~50 lines of build code plus a non-obvious Def.sequential snapshot dance to defeat .value-macro hoisting. It works, but it's exactly the kind of thing the API should make unnecessary.

Proposed shape

Two complementary additions, both backwards-compatible:

sealed trait MatrixAxis
object MatrixAxis {
  // Today's behavior: literal values, each rendered through `wrap`.
  final case class Values(values: List[String]) extends MatrixAxis
  // The whole axis IS a raw GHA expression that evaluates to an array.
  final case class FromExpression(expr: String) extends MatrixAxis
}

// Existing API unchanged:
def matrixAdds: Map[String, List[String]]   // -> MatrixAxis.Values internally

// New, additive:
def matrixAxes: Map[String, MatrixAxis]
def withMatrixAxes(axes: Map[String, MatrixAxis]): WorkflowJob

User code becomes:

WorkflowJob(
  id = "scripted",
  needs = List("scripted-discover"),
  matrixAxes = Map(
    "test" -> MatrixAxis.FromExpression("fromJSON(needs.scripted-discover.outputs.tests)"),
  ),
  // ...
)

A symmetric fix for outputs: is worth considering — same Literal | Expression distinction would eliminate the double-wrap footgun — but the matrix axis is the only place where the current API genuinely cannot express what GHA supports, so that's the priority.

Why it's worth doing

  • The "discover at CI time, fan out via matrix" pattern isn't exotic — it's the standard answer for "I have a variable-length set of independent test shards" in GitHub Actions, and several large projects (smithy4s among them) hand-write their workflows partly because of this gap.
  • The fix is small, additive, and fits the existing Impl-private convention so MiMa filters aren't a blocker.
  • It removes a class of fragile build-script patterns that future readers have to reverse-engineer.

Happy to put up a PR if the design lands.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at WorkflowJob.matrixAdds and the renderer's wrap helper, then trace githubWorkflowGenerate and githubWorkflowCheck to understand how matrix values are emitted and checked. Implement the additive MatrixAxis and matrixAxes entry points so FromExpression produces an unquoted axis expression while existing matrixAdds behavior remains unchanged; verify that generated workflows preserve the runtime-discovered matrix shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, scala
Domain
build-system, ci-cd
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.