actions / actions/runner-container-hooks

Workflow pod forced on the same node in Kubernetes mode, even with RWX volume

Open
#227 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
153
Forks
112
Avg merge
6m
Merged PRs (30d)
1

Description

Checks
Controller Version

0.12.1

Deployment Method

Helm

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Deploy a runnerset in Kubernetes mode, using a ReadWriteMany volume type for `kubernetesModeWorkVolumeClaim`
2. Set the env variable `ACTIONS_RUNNER_USE_KUBE_SCHEDULER` to "true" on the runner pod to enable separate scheduling of the workflow pod
3. Trigger a workflow using this runner and look at the manifest of the created workflow pod
Describe the bug

In Kubernetes mode, a nodeAffinity is set on the workflow pod so that it gets scheduled on the same node than the runner :

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - <runnerNodeName>

This is not a wanted behavior when we are using a ReadWriteMany volume for the "_work" volume, because we expect the runner and workflow pods to be able to schedule on different nodes depending on their resource requests.

Describe the expected behavior

The workflow pod is handled by the Kubernetes scheduler separately from the runner pod and can be placed on whatever node has enough resources available. The work volume is shared between both pods even on different nodes, thanks to the ReadWriteMany mode.

Additional Context
containerMode:
    type: "kubernetes"
    kubernetesModeWorkVolumeClaim:
      accessModes: ["ReadWriteMany"]
      storageClassName: <azure-file-nfs-sc>
      resources:
        requests:
          storage: 1Ti

  template:
    spec:
      containers:
      - name: runner
        image: <container-image>
        command: ["/home/runner/run.sh"]
        securityContext:
          privileged: true
        env:
        - name: ACTIONS_RUNNER_USE_KUBE_SCHEDULER
          value: "true"
Controller Logs
Not relevant
Runner Pod Logs
Not relevant

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the workflow-pod scheduling path controlled by ACTIONS_RUNNER_USE_KUBE_SCHEDULER and inspect the generated pod manifest. Reproduce with the provided Helm runnerset configuration and a ReadWriteMany work volume. Done means the workflow pod is not forced to the runner node when separate Kubernetes scheduling is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, kubernetes, typescript
Domain
ci-cd, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.