rapidsai / rapidsai/shared-workflows

Implement "dispatch" pattern from telemetry shared-actions

Open
#378 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request improvement Needs build-infra non-breaking
Dominant language
Shell
Stars
8
Forks
33
Avg merge
23h 51m
Merged PRs (30d)
19

Description

One thing useful that came out of the telemetry work was the "dispatch" pattern, which is described in the shared-actions repo readme. The general idea is that instead of calling an action directly with a repo/branch reference, you should first manually clone the shared-actions repo at that branch, and then call the desired action with a relative path. The advantages to doing this are:

  • It's much easier to include code outside of actions, such as native Python files. When a shared action is called directly, any other files in the repo do not come along for the ride. Using native files facilitates use of linters and other code helper tools.
  • It's much easier to change branch references for several actions at once. In the shared-actions workflow, we run a telemetry-stash-base-env-vars action that captures several environment variables, two of which are the SHARED_ACTIONS_REPO and SHARED_ACTIONS_REF variables. This are written to a file, and stored as a github artifact. Subsequent actions (which use very stable logic on the main branch) load that file, read those env vars, clone the requested repo/ref, and call the requested shared action with relative path.
  • This is a hack around the restricted list of actions that rapids jobs can clone, allowing us to point to our user forks instead of pushing branches directly to the rapidsai org repo.

For example, instead of changing multiple branch references to test a shared-workflows branch, like:

Image

One could have:

# This is a project's pr.yaml

jobs:
  shared-vars-setup:
    runs-on: ubuntu-latest
    continue-on-error: true
     env:
        SHARED_WORKFLOWS_REPO: rapidsai/shared-workflows
        SHARED_WORKFLOWS_ACTION: cuda-12.9.0
      steps:
         - name: Stash shared variables setup
           uses: rapidsai/shared-actions/telemetry-dispatch-stash-base-env-vars@main
  cpp-build:
    needs: [shared-vars-setup]
    secrets: inherit
    uses: rapidsai/shared-workflows/.github/workflows/dispatch-conda-cpp-build.yaml@main
    with:
      build_type: ${{ inputs.build_type || 'branch' }}
      branch: ${{ inputs.branch }}

and dispatch-conda-cpp-build.yaml would be:

inputs:
  build_type: ...
  branch: ...

jobs:
  shared-vars-load-or-default:
        steps:
          # Downloads the env var file, loads it, then clones shared-workflows according to env var values
          - name: Load shared variables setup
            uses: rapidsai/shared-actions/load-env-vars-then-clone-workflows@main
   cpp-build:
        needs: [shared-vars-load-or-default]
        uses: ./shared-workflows/.github/workflows/conda-cpp-build.yaml
        with:
            build_type: ${{inputs.build_type}}
            branch: ${{inputs.branch}}

This adds overhead for the indirection, but when there are multiple shared-workflows being called, the developer need only set the values in the base parameter setup:

# This is a project's pr.yaml

jobs:
  shared-vars-setup:
    runs-on: ubuntu-latest
     env:
        SHARED_WORKFLOWS_REPO: rapidsai/shared-workflows
        SHARED_WORKFLOWS_ACTION: cuda-12.9.0
      steps:
        - name: Stash shared variables setup
          uses: rapidsai/shared-actions/telemetry-dispatch-stash-base-env-vars@main

All references to @main or @branch-25.08 in the project's pr.yaml file would not need to change, unless you needed to point to a new or altered dispatch action/workflow.

Implementation note:

Stashing and loading shared variables can utilize the existing shared-actions code. We would need to generalize the clone or copy the script to differentiate destination path of the cloned folder.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the shared-actions README and its existing code for stashing/loading shared variables and cloning repositories. Define the dispatch workflow changes around the examples in this issue, then verify that shared workflows can clone the requested repository and invoke actions or workflows through relative paths without changing each reference.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github-actions, shell
Domain
ci-cd, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.