actions / actions/actions-runner-controller

Runners not exposing correct available system resources when Docker context is used

Open
#2,748 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug community needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

v0.27.2

Helm Chart Version

0.23.3

CertManager Version

No response

Deployment Method

Helm

cert-manager installation

yes

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    meta.helm.sh/release-name: staging-4c-16gb
    meta.helm.sh/release-namespace: staging-4c-16gb
  creationTimestamp: "2022-10-26T17:57:42Z"
  generation: 789940
  labels:
    app.kubernetes.io/managed-by: Helm
  name: gha-staging-4c-16gb
  namespace: staging-4c-16gb
  resourceVersion: "232685229"
  uid: 9520fafe-3d96-49cb-8570-ff3f6b726307
spec:
  effectiveTime: "2023-07-17T10:13:25Z"
  replicas: 9
  selector: null
  template:
    metadata: {}
    spec:
      dockerVolumeMounts:
      - mountPath: /var/lib/docker
        name: docker-extra
      dockerdWithinRunnerContainer: true
      dockerdContainerResources:
        limits:
          cpu: "4.0"
          memory: "8Gi"
        requests:
          cpu: "2.0"
          memory: "4Gi"
      env:
      - name: RUNNER_FEATURE_FLAG_EPHEMERAL
        value: "true"
      - name: DISABLE_RUNNER_UPDATE
        value: "false"
      group: sp-integration-ubuntu-4c-16gb-prod-weu
      image: xxxxx/github-runner-full:latest-stable
      imagePullSecrets:
      - name: artifactory-imagepullsecret
      initContainers:
      - command:
        - sh
        - -c
        - chmod -R 606 /dev/kvm
        image: busybox
        name: kvm-permission
        resources: {}
        volumeMounts:
        - mountPath: /dev/kvm
          name: kvm-device
      labels:
      - staging-4c-16gb
      - self-hosted
      nodeSelector:
        agentpool: stgrunneri
      organization: xx
      resources:
        limits:
          cpu: "4"
          memory: 16Gi
        requests:
          cpu: "4"
          memory: 16Gi
      sidecarContainers:
      - command:
        - bash
        - -c
        - --
        - bash -c 'while true; do tail -f /runner/_diag/*.log; [ $? -ne 0 ] && sleep
          3; done;'
        image: bash:latest
        name: logger
        resources: {}
        volumeMounts:
        - mountPath: /runner/_diag
          name: runnerlogs
      volumeMounts:
      - mountPath: /runner/_diag
        name: runnerlogs
      - mountPath: /dev/kvm
        name: kvm-device
      volumes:
      - hostPath:
          path: /mnt/docker-extra
          type: DirectoryOrCreate
        name: docker-extra
      - hostPath:
          path: /dev/kvm
        name: kvm-device
      - emptyDir: {}
        name: runnerlogs
status:
  availableReplicas: 9
  desiredReplicas: 9
  readyReplicas: 9
  replicas: 9
  updatedReplicas: 9
To Reproduce
---
name: Test CPU information
on:
  workflow_dispatch:
  push:
    branches:
      - test-cpu

jobs:
  test:
    runs-on: ["self-hosted", "staging-4c-16gb"]
    steps:
      - name: Print CPU info
        run: |
          # echo "### lscpu ###"
          # lscpu
          # echo "-----"
          echo "### /sys/fs/cgroup/cpu/cpu.cfs_quota_us + /sys/fs/cgroup/cpu/cpu.cfs_period_us"
          cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
          cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
  test-dind:
    runs-on: ["self-hosted", "staging-4c-16gb"]
    container: "ubuntu"
    steps:
      - name: Print CPU info in dind
        run: |
          # echo "### lscpu ###"
          # lscpu
          # echo "-----"
          echo "### /sys/fs/cgroup/cpu/cpu.cfs_quota_us + /sys/fs/cgroup/cpu/cpu.cfs_period_us"
          cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
          cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
Describe the bug

Run # echo "### lscpu ###"

/sys/fs/cgroup/cpu/cpu.cfs_quota_us + /sys/fs/cgroup/cpu/cpu.cfs_period_us

-1
100000

Describe the expected behavior

Run # echo "### lscpu ###"

/sys/fs/cgroup/cpu/cpu.cfs_quota_us + /sys/fs/cgroup/cpu/cpu.cfs_period_us

400000
100000

Whole Controller Logs
*
Whole Runner Pod Logs
*
Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the RunnerDeployment configuration and reproduction workflow in the issue, focusing on dockerdWithinRunnerContainer and the cgroup CPU quota files. Reproduce the reported mismatch for a runner and for a job container, then trace how Docker context affects exposed CPU limits. Done means the reported quota reflects the configured 4 CPUs rather than -1.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.