actions / actions/actions-runner-controller

Multitenancy with `singleNamespace: true` does not work

Open
#2,579 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

0.27.4

Helm Chart Version

0.23.2

CertManager Version

1.7.1

Deployment Method

Helm

cert-manager installation
helm upgrade cert-manager cert-manager --repo https://charts.jetstack.io \
  --namespace cert-manager --version v1.7.1 \
  -f values.yaml --install
Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
apiVersion: v1
kind: Secret
metadata:
  name: actions-runner-controller-org-b
  namespace: actions-runner
type: Opaque
data:
  github_app_id: APP_ID
  github_app_installation_id: INSTALLATION_ID
  github_app_private_key:  PRIVATE_KEY
  github_webhook_secret_token: TOKEN
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-b
  namespace: actions-runner
spec:
  template:
    spec:
      organization: org-b
      githubAPICredentialsFrom:
        secretRef:
          name: actions-runner-controller-org-b
      labels:
        - self-hosted

      env:
        # Disables automatic runner updates
        - name: DISABLE_RUNNER_UPDATE
          value: "true"
        # In case of spot interruption
        - name: RUNNER_GRACEFUL_STOP_TIMEOUT
          value: "100"

      serviceAccountName: github-action-runner

      dockerdWithinRunnerContainer: true
      image: our-custom-dind-image
      imagePullPolicy: Always
      dockerRegistryMirror: INTERNAL_MIRROR

      resources:
        requests:
          cpu: "2"
          memory: "4Gi"
        limits:
          cpu: "2"
          memory: "4Gi"
          # ephemeral-storage: "64Gi"

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: role
                    operator: In
                    values:
                      - actions-runner
                  - key: instance-lifecycle
                    operator: NotIn
                    values:
                    - ondemand
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                namespaces:
                  - actions-runner
                labelSelector:
                  matchLabels:
                    app: github-actions-runner
              weight: 1
            - podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                namespaces:
                  - actions-runner
                labelSelector:
                  matchLabels:
                    component: 2core-4gb
              weight: 10

      tolerations:
      - key: dedicated
        value: actions-runner
        effect: NoSchedule
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: runner-b
  namespace: actions-runner
spec:
  githubAPICredentialsFrom:
    secretRef:
      name: actions-runner-controller-org-b
  scaleDownDelaySecondsAfterScaleOut: 240
  scaleTargetRef:
    name: runner-b
  minReplicas: 0
  maxReplicas: 20
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'    # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up
    scaleDownThreshold: '0.3'   # The percentage of busy runners at which the number of desired runners are re-evaluated to scale down
    scaleUpFactor: '1.4'        # The scale up multiplier factor applied to desired count
    scaleDownFactor: '0.5'      # The scale down multiplier factor applied to desired count
  scaleUpTriggers:
  - githubEvent:
      workflowJob: {}
    amount: 1
    duration: "1m"
To Reproduce
1. Deploy actions-runner-controller in `actions-runner-system` namespace, with `singleNamespace: true`
2. Deploy runner deployment in `actions-runner` namespace (`runner-a`)
3. Add runner deployment with different github organization (`runner-b`) from step 2 using `githubAPICredentialsFrom.secretRef`. In this step, secret for different org should be placed in `actions-runner` namespace
    - https://github.com/actions/actions-runner-controller/blob/master/docs/using-arc-across-organizations.md
4. Increase runner replica of runner deployment `runner-b`, pod is not created and controller error log is written as below.
Describe the bug

As #2265 removed rbac permission on secrets from manager cluster role, and make it 'Role', not 'ClusterRole' when singleNamespace: true(https://github.com/actions/actions-runner-controller/pull/2265/files#diff-f56843013db5093a61ddbca489859572952535b805454ff25c458b17fe0c50c2R3), controller no longer has permission on secrets in actions-runner namespace.

Describe the expected behavior

Manager has clusterrole so that it has permission on secrets in actions-runner namespace

Whole Controller Logs
{"severity":"error","ts":"2023-05-12T06:41:54Z","message":"Reconciler error","controller":"horizontalrunnerautoscaler-controller","controllerGroup":"actions.summerwind.dev","controllerKind":"HorizontalRunnerAutoscaler","HorizontalRunnerAutoscaler":{"name":"RUNNER_NAME","namespace":"actions-runner"},"namespace":"actions-runner","name":"RUNNER_NAME","reconcileID":"30f761d2-4c19-4c59-9baa-55733ca414a3","error":"secrets \"actions-runner-controller-org-b\" is forbidden: User \"system:serviceaccount:actions-runner-system:actions-runner-controller\" cannot get resource \"secrets\" in API group \"\" in the namespace \"actions-runner\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
Whole Runner Pod Logs
no pod logs because it's not created.
Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the singleNamespace RBAC change referenced in PR #2265 and the cross-organization setup in docs/using-arc-across-organizations.md. Reproduce the HorizontalRunnerAutoscaler failure and inspect the controller's permissions for secrets in the actions-runner namespace. Done means runner-b is reconciled successfully and its pod is created without the forbidden secret error.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, helm, kubernetes
Domain
devops, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.