actions / actions/actions-runner-controller
Runners not exposing correct available system resources when Docker context is used
Open
Nobody has claimed this yet.
bug
community
needs triage
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 1.5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 27
Description
Checks
- I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
- I'm not using a custom entrypoint in my runner image
Controller Version
v0.27.2
Helm Chart Version
0.23.3
CertManager Version
No response
Deployment Method
Helm
cert-manager installation
yes
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
- I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
- My actions-runner-controller version (v0.x.y) does support the feature
- I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
- I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
annotations:
meta.helm.sh/release-name: staging-4c-16gb
meta.helm.sh/release-namespace: staging-4c-16gb
creationTimestamp: "2022-10-26T17:57:42Z"
generation: 789940
labels:
app.kubernetes.io/managed-by: Helm
name: gha-staging-4c-16gb
namespace: staging-4c-16gb
resourceVersion: "232685229"
uid: 9520fafe-3d96-49cb-8570-ff3f6b726307
spec:
effectiveTime: "2023-07-17T10:13:25Z"
replicas: 9
selector: null
template:
metadata: {}
spec:
dockerVolumeMounts:
- mountPath: /var/lib/docker
name: docker-extra
dockerdWithinRunnerContainer: true
dockerdContainerResources:
limits:
cpu: "4.0"
memory: "8Gi"
requests:
cpu: "2.0"
memory: "4Gi"
env:
- name: RUNNER_FEATURE_FLAG_EPHEMERAL
value: "true"
- name: DISABLE_RUNNER_UPDATE
value: "false"
group: sp-integration-ubuntu-4c-16gb-prod-weu
image: xxxxx/github-runner-full:latest-stable
imagePullSecrets:
- name: artifactory-imagepullsecret
initContainers:
- command:
- sh
- -c
- chmod -R 606 /dev/kvm
image: busybox
name: kvm-permission
resources: {}
volumeMounts:
- mountPath: /dev/kvm
name: kvm-device
labels:
- staging-4c-16gb
- self-hosted
nodeSelector:
agentpool: stgrunneri
organization: xx
resources:
limits:
cpu: "4"
memory: 16Gi
requests:
cpu: "4"
memory: 16Gi
sidecarContainers:
- command:
- bash
- -c
- --
- bash -c 'while true; do tail -f /runner/_diag/*.log; [ $? -ne 0 ] && sleep
3; done;'
image: bash:latest
name: logger
resources: {}
volumeMounts:
- mountPath: /runner/_diag
name: runnerlogs
volumeMounts:
- mountPath: /runner/_diag
name: runnerlogs
- mountPath: /dev/kvm
name: kvm-device
volumes:
- hostPath:
path: /mnt/docker-extra
type: DirectoryOrCreate
name: docker-extra
- hostPath:
path: /dev/kvm
name: kvm-device
- emptyDir: {}
name: runnerlogs
status:
availableReplicas: 9
desiredReplicas: 9
readyReplicas: 9
replicas: 9
updatedReplicas: 9
To Reproduce
---
name: Test CPU information
on:
workflow_dispatch:
push:
branches:
- test-cpu
jobs:
test:
runs-on: ["self-hosted", "staging-4c-16gb"]
steps:
- name: Print CPU info
run: |
# echo "### lscpu ###"
# lscpu
# echo "-----"
echo "### /sys/fs/cgroup/cpu/cpu.cfs_quota_us + /sys/fs/cgroup/cpu/cpu.cfs_period_us"
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
test-dind:
runs-on: ["self-hosted", "staging-4c-16gb"]
container: "ubuntu"
steps:
- name: Print CPU info in dind
run: |
# echo "### lscpu ###"
# lscpu
# echo "-----"
echo "### /sys/fs/cgroup/cpu/cpu.cfs_quota_us + /sys/fs/cgroup/cpu/cpu.cfs_period_us"
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
Describe the bug
Run # echo "### lscpu ###"
/sys/fs/cgroup/cpu/cpu.cfs_quota_us + /sys/fs/cgroup/cpu/cpu.cfs_period_us
-1
100000
Describe the expected behavior
Run # echo "### lscpu ###"
/sys/fs/cgroup/cpu/cpu.cfs_quota_us + /sys/fs/cgroup/cpu/cpu.cfs_period_us
400000
100000
Whole Controller Logs
*
Whole Runner Pod Logs
*
Additional Context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the RunnerDeployment configuration and reproduction workflow in the issue, focusing on dockerdWithinRunnerContainer and the cgroup CPU quota files. Reproduce the reported mismatch for a runner and for a job container, then trace how Docker context affects exposed CPU limits. Done means the reported quota reflects the configured 4 CPUs rather than -1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100