actions / actions/runner

Job Interference using Reusable Workflows with Matrix Strategy

Open
#2,475 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
Using reusable workflows with matrix strategies is now supported (thank you!). However, there seems to be a flakey bug where the jobs created by the reusable workflow interact in different ways. Let's say there is a matrix strategy creating two reusable workflow calls, BUILD_1 and BUILD_2. Then,

  • Sometimes the job JOB_A for BUILD_1 appears in BUILD_2

Both of JOB_A's had the same full name (so they were both referencing the same matrix strategy input)

https://github.com/project-oak/oak/actions/runs/4347726368/jobs/7604490843
https://github.com/project-oak/oak/actions/runs/4347726368/jobs/7604492489

BUT their inputs to the steps where different and corresponded to the different invocations of the reusable workflow.

This was caused when clicking the UI for re-running the workflow when it failed.

Are these reusable workflow invocations truly isolated and running in different environments? We are particularly concerned because our project requires isolation in each reusable workflow environment in order to produce a build that satisfies isolation properties from other builds.

To Reproduce
Steps to reproduce the behavior:

  1. Use a reusable workflow with matrix strategy, where one job requires another. Say, one job uploads an artifact and the second downloads.
jobs:
  build_binary:
    # We use the same job template to generate provenances for multiple binaries.
    strategy:
      fail-fast: false
      matrix:
        buildconfig:
          - buildconfigs/slsav1_oak_functions_enclave_app.toml
          - buildconfigs/slsav1_oak_tensorflow_enclave_app.toml

    permissions:
      actions: read
      id-token: write
      contents: write
      pull-requests: write
    uses: ./.github/workflows/reusable_provenance.yaml
    with:
      build-config-path: ${{ matrix.buildconfig }}

Expected behavior
We expect that each BUILD runs in separate environments, and the JOBs do not interact

Runner Version and Platform

Image: ubuntu-20.04
Version: 20230224.2

OS of the machine running the runner? Linux

What's not working?

A job appears twice in one build, but not in the other:
unnamed

Job Log Output

Runner and Worker's Diagnostic Logs

https://github.com/project-oak/oak/actions/runs/4347726368/jobs/7604490843
https://github.com/project-oak/oak/actions/runs/4347726368/jobs/7604492489

cc @rbehjati @laurentsimon @ianlewis @jhutchings1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reusable_provenance.yaml workflow and its matrix buildconfig example, then inspect the linked job logs and runner diagnostic logs. Reproduce the problem by rerunning the workflow and compare the two reusable-workflow invocations. Done means matrix jobs remain isolated, with each job showing only the inputs and artifacts from its own build.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, github-actions
Domain
ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.