actions / actions/actions-runner-controller
Multitenancy with `singleNamespace: true` does not work
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 1.5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 27
Description
Checks
- I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
- I'm not using a custom entrypoint in my runner image
Controller Version
0.27.4
Helm Chart Version
0.23.2
CertManager Version
1.7.1
Deployment Method
Helm
cert-manager installation
helm upgrade cert-manager cert-manager --repo https://charts.jetstack.io \
--namespace cert-manager --version v1.7.1 \
-f values.yaml --install
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
- I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
- My actions-runner-controller version (v0.x.y) does support the feature
- I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
- I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
apiVersion: v1
kind: Secret
metadata:
name: actions-runner-controller-org-b
namespace: actions-runner
type: Opaque
data:
github_app_id: APP_ID
github_app_installation_id: INSTALLATION_ID
github_app_private_key: PRIVATE_KEY
github_webhook_secret_token: TOKEN
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: runner-b
namespace: actions-runner
spec:
template:
spec:
organization: org-b
githubAPICredentialsFrom:
secretRef:
name: actions-runner-controller-org-b
labels:
- self-hosted
env:
# Disables automatic runner updates
- name: DISABLE_RUNNER_UPDATE
value: "true"
# In case of spot interruption
- name: RUNNER_GRACEFUL_STOP_TIMEOUT
value: "100"
serviceAccountName: github-action-runner
dockerdWithinRunnerContainer: true
image: our-custom-dind-image
imagePullPolicy: Always
dockerRegistryMirror: INTERNAL_MIRROR
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
cpu: "2"
memory: "4Gi"
# ephemeral-storage: "64Gi"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: role
operator: In
values:
- actions-runner
- key: instance-lifecycle
operator: NotIn
values:
- ondemand
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
topologyKey: kubernetes.io/hostname
namespaces:
- actions-runner
labelSelector:
matchLabels:
app: github-actions-runner
weight: 1
- podAffinityTerm:
topologyKey: kubernetes.io/hostname
namespaces:
- actions-runner
labelSelector:
matchLabels:
component: 2core-4gb
weight: 10
tolerations:
- key: dedicated
value: actions-runner
effect: NoSchedule
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
name: runner-b
namespace: actions-runner
spec:
githubAPICredentialsFrom:
secretRef:
name: actions-runner-controller-org-b
scaleDownDelaySecondsAfterScaleOut: 240
scaleTargetRef:
name: runner-b
minReplicas: 0
maxReplicas: 20
metrics:
- type: PercentageRunnersBusy
scaleUpThreshold: '0.75' # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up
scaleDownThreshold: '0.3' # The percentage of busy runners at which the number of desired runners are re-evaluated to scale down
scaleUpFactor: '1.4' # The scale up multiplier factor applied to desired count
scaleDownFactor: '0.5' # The scale down multiplier factor applied to desired count
scaleUpTriggers:
- githubEvent:
workflowJob: {}
amount: 1
duration: "1m"
To Reproduce
1. Deploy actions-runner-controller in `actions-runner-system` namespace, with `singleNamespace: true`
2. Deploy runner deployment in `actions-runner` namespace (`runner-a`)
3. Add runner deployment with different github organization (`runner-b`) from step 2 using `githubAPICredentialsFrom.secretRef`. In this step, secret for different org should be placed in `actions-runner` namespace
- https://github.com/actions/actions-runner-controller/blob/master/docs/using-arc-across-organizations.md
4. Increase runner replica of runner deployment `runner-b`, pod is not created and controller error log is written as below.
Describe the bug
As #2265 removed rbac permission on secrets from manager cluster role, and make it 'Role', not 'ClusterRole' when singleNamespace: true(https://github.com/actions/actions-runner-controller/pull/2265/files#diff-f56843013db5093a61ddbca489859572952535b805454ff25c458b17fe0c50c2R3), controller no longer has permission on secrets in actions-runner namespace.
Describe the expected behavior
Manager has clusterrole so that it has permission on secrets in actions-runner namespace
Whole Controller Logs
{"severity":"error","ts":"2023-05-12T06:41:54Z","message":"Reconciler error","controller":"horizontalrunnerautoscaler-controller","controllerGroup":"actions.summerwind.dev","controllerKind":"HorizontalRunnerAutoscaler","HorizontalRunnerAutoscaler":{"name":"RUNNER_NAME","namespace":"actions-runner"},"namespace":"actions-runner","name":"RUNNER_NAME","reconcileID":"30f761d2-4c19-4c59-9baa-55733ca414a3","error":"secrets \"actions-runner-controller-org-b\" is forbidden: User \"system:serviceaccount:actions-runner-system:actions-runner-controller\" cannot get resource \"secrets\" in API group \"\" in the namespace \"actions-runner\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
Whole Runner Pod Logs
no pod logs because it's not created.
Additional Context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the singleNamespace RBAC change referenced in PR #2265 and the cross-organization setup in docs/using-arc-across-organizations.md. Reproduce the HorizontalRunnerAutoscaler failure and inspect the controller's permissions for secrets in the actions-runner namespace. Done means runner-b is reconciled successfully and its pod is created without the forbidden secret error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go, helm, kubernetes
- Domain
- devops, infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100