docker / docker/bake-action

[subaction/matrix] Generate multiple matrices when targets reference other targets as build contexts

Open
#435 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhancement
Dominant language
TypeScript
Stars
301
Forks
43
Avg merge
2d 5h
Merged PRs (30d)
20

Description

Description

We'd like to use the matrix subaction to generate multiple matrices, since we have a couple of targets that depend on other targets and would like to avoid building the same target multiple times.


As an example, consider the following docker-bake.hcl (containing targets with Dockerfiles that can't be easily merged into a multi-stage Dockerfile):

group "default" {
    targets = ["apps", "special-app"]
}

target "base" {
    context = "basedir"
    tags = ["base:latest"]
}

target "apps" {
    name = "app_${version}"
    context = "appdir/${version}"
    matrix =  {
        version = ["1-0-0", "2-0-0", "2-1-5"]
    }
    tags = ["app:${version}"]
    contexts = {
        base = "target:base"
    }
}

target "special-app" {
    context = "appdir/special"
    tags = ["special-app:latest"]
    contexts = {
        app = "target:app_2-0-0"
    }
}

Now we commit and push a change to basedir/Dockerfile, which triggers a GitHub workflow that builds these targets on multiple runners using the matrix subaction and pushes them to our registry.

Currently the subaction generates the following matrix:

[{"target": "base"}, {"target": "app_1-0-0"}, {"target": "app_2-0-0"}, {"target": "app_2-1-5"}, {"target": "special-app"}]

This generates 5 workflow jobs running in parallel, but most of them spend time building dependent targets first, which other jobs are already building. This results in a lot of duplicate work which could be avoided.

Ideally the subaction would generate multiple matrices like this:

[
    [{"target": "base"}],
    [{"target": "app_1-0-0"}, {"target": "app_2-0-0"}, {"target": "app_2-1-5"}],
    [{"target": "special-app"}]
]

This way we could define multiple jobs that depend on the previous ones and supply them with one matrix each.

Snippet of an example workflow using multiple matrices
jobs:
  generate:
    runs-on: ubuntu-latest
    outputs:
      jobs_amount: ${{ steps.generate.outputs.jobs_amount }}
      matrices: ${{ steps.generate.outputs.matrices }}
    steps:
      - uses: actions/checkout@v6
      - id: generate
        uses: docker/bake-action/subaction/matrix@multiple-matrices

  stage_1:
    needs:
      - generate
    if: ${{ fromJson(needs.generate.outputs.jobs_amount) >= 1 }}
    strategy:
      matrix:
        include: ${{ toJson(fromJson(needs.generate.outputs.matrices)[0]) }}
    steps:
      # ...

  stage_2:
    needs:
      - generate
      - stage_1
    if: ${{ fromJson(needs.generate.outputs.jobs_amount) >= 2 }}
    strategy:
      matrix:
        include: ${{ toJson(fromJson(needs.generate.outputs.matrices)[1]) }}
    steps:
      # ...

  stage_3:
    needs:
      - generate
      - stage_2
    if: ${{ fromJson(needs.generate.outputs.jobs_amount) >= 3 }}
    strategy:
      matrix:
        include: ${{ toJson(fromJson(needs.generate.outputs.matrices)[2]) }}
    steps:
      # ...

We imagine that this usecase is common enough to be supported here and would like to contribute that feature, since we already made it for our own workflows.

Would you be interested in this feature?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the matrix subaction and trace how the current target list is generated from the docker-bake.hcl example. Define completion by producing dependency-grouped matrices plus jobs_amount and matrices outputs that the staged GitHub Actions workflow can consume without duplicating dependent builds.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, typescript
Domain
ci-cd, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.