[subaction/matrix] Generate multiple matrices when targets reference other targets as build contexts
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 301
- Forks
- 43
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 20
Description
Description
We'd like to use the matrix subaction to generate multiple matrices, since we have a couple of targets that depend on other targets and would like to avoid building the same target multiple times.
As an example, consider the following docker-bake.hcl (containing targets with Dockerfiles that can't be easily merged into a multi-stage Dockerfile):
group "default" {
targets = ["apps", "special-app"]
}
target "base" {
context = "basedir"
tags = ["base:latest"]
}
target "apps" {
name = "app_${version}"
context = "appdir/${version}"
matrix = {
version = ["1-0-0", "2-0-0", "2-1-5"]
}
tags = ["app:${version}"]
contexts = {
base = "target:base"
}
}
target "special-app" {
context = "appdir/special"
tags = ["special-app:latest"]
contexts = {
app = "target:app_2-0-0"
}
}
Now we commit and push a change to basedir/Dockerfile, which triggers a GitHub workflow that builds these targets on multiple runners using the matrix subaction and pushes them to our registry.
Currently the subaction generates the following matrix:
[{"target": "base"}, {"target": "app_1-0-0"}, {"target": "app_2-0-0"}, {"target": "app_2-1-5"}, {"target": "special-app"}]
This generates 5 workflow jobs running in parallel, but most of them spend time building dependent targets first, which other jobs are already building. This results in a lot of duplicate work which could be avoided.
Ideally the subaction would generate multiple matrices like this:
[
[{"target": "base"}],
[{"target": "app_1-0-0"}, {"target": "app_2-0-0"}, {"target": "app_2-1-5"}],
[{"target": "special-app"}]
]
This way we could define multiple jobs that depend on the previous ones and supply them with one matrix each.
Snippet of an example workflow using multiple matrices
jobs:
generate:
runs-on: ubuntu-latest
outputs:
jobs_amount: ${{ steps.generate.outputs.jobs_amount }}
matrices: ${{ steps.generate.outputs.matrices }}
steps:
- uses: actions/checkout@v6
- id: generate
uses: docker/bake-action/subaction/matrix@multiple-matrices
stage_1:
needs:
- generate
if: ${{ fromJson(needs.generate.outputs.jobs_amount) >= 1 }}
strategy:
matrix:
include: ${{ toJson(fromJson(needs.generate.outputs.matrices)[0]) }}
steps:
# ...
stage_2:
needs:
- generate
- stage_1
if: ${{ fromJson(needs.generate.outputs.jobs_amount) >= 2 }}
strategy:
matrix:
include: ${{ toJson(fromJson(needs.generate.outputs.matrices)[1]) }}
steps:
# ...
stage_3:
needs:
- generate
- stage_2
if: ${{ fromJson(needs.generate.outputs.jobs_amount) >= 3 }}
strategy:
matrix:
include: ${{ toJson(fromJson(needs.generate.outputs.matrices)[2]) }}
steps:
# ...
We imagine that this usecase is common enough to be supported here and would like to contribute that feature, since we already made it for our own workflows.
Would you be interested in this feature?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the matrix subaction and trace how the current target list is generated from the docker-bake.hcl example. Define completion by producing dependency-grouped matrices plus jobs_amount and matrices outputs that the staged GitHub Actions workflow can consume without duplicating dependent builds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, typescript
- Domain
- ci-cd, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100