actions / actions/actions-runner-controller

Helm chart references wrong secret for metrics-server

Open
#2,175 4 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

0.27.0

Helm Chart Version

0.22.0

CertManager Version

No response

Deployment Method

Helm

cert-manager installation

Yes

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
Chart values:

    actionsMetricsServer:
      enabled: true
      secret:
        enabled: true
      service:
        ports:
        - name: http
          port: 8000
          protocol: TCP
          targetPort: http
      serviceMonitor: true
    authSecret:
      create: false # Manually create 'controller-manager' secret with GH creds
    githubWebhookServer:
      enabled: true
      podDisruptionBudget:
        enabled: true
        minAvailable: 1
      replicaCount: 2
    metrics:
      proxy:
        enabled: false
      serviceMonitor: true


### To Reproduce

```markdown
Enable metrics-server, check the env vars
Describe the bug

The helm chart is trying to pull Github credentials (GITHUB_TOKEN, GITHUB_APP_ID etc) from the secret storing the webhook secret rather than the secret storing credentials.

https://github.com/actions/actions-runner-controller/blob/0324658a3fad79237b5da9c21a3ce6f6cc9a4429/charts/actions-runner-controller/templates/actionsmetrics.deployment.yaml#L72-L96

> k describe deploy actions-runner-controller-actions-metrics-server | grep GITHUB_APP_ID
      GITHUB_APP_ID:                <set to the key 'github_app_id' in secret 'github-webhook-server'>                 Optional: true

> k describe deploy actions-runner-controller | grep GITHUB_APP_ID
      GITHUB_APP_ID:               <set to the key 'github_app_id' in secret 'controller-manager'>               Optional: true

Credentials are in the (default name) controller-manager secret, I am not using a webhook secret at all currently so github-webhook-server doesn't exist.

> k get secret | grep Opaque
controller-manager                                            Opaque                                3      615d

The GH client never gets initialised and then we get a segfault when a webhook is received.

2023-01-17T21:09:18Z	INFO	GitHub client is not initialized. Runner groups with custom visibility are not supported. If needed, please provide GitHub authentication. This will incur in extra GitHub API calls

2023-01-17T21:09:36Z	INFO	workflowjobmetrics-webhookserver	ok	{"event": "workflow_job", "hookID": "397065108", "delivery": "78d6e540-969b-11ed-90bd-fb42ff6449d0"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1344519]

goroutine 64 [running]:
github.com/actions/actions-runner-controller/pkg/actionsmetrics.(*EventReader).fetchAndParseWorkflowJobLogs(0x14ba860?, {0x18c4650?, 0xc00004e040?}, 0xe?)
	github.com/actions/actions-runner-controller/pkg/actionsmetrics/event_reader.go:167 +0x59
github.com/actions/actions-runner-controller/pkg/actionsmetrics.(*EventReader).ProcessWorkflowJobEvent(0xc000590030, {0x18c4650, 0xc00004e040}, {0x152af80?, 0xc00001b590})
	github.com/actions/actions-runner-controller/pkg/actionsmetrics/event_reader.go:99 +0x6e5
github.com/actions/actions-runner-controller/pkg/actionsmetrics.(*EventReader).ProcessWorkflowJobEvents(0xc000590030, {0x18c4650, 0xc00004e040})
	github.com/actions/actions-runner-controller/pkg/actionsmetrics/event_reader.go:44 +0x59
main.main.func1()
	github.com/actions/actions-runner-controller/cmd/actionsmetricsserver/main.go:157 +0x6c
created by main.main
	github.com/actions/actions-runner-controller/cmd/actionsmetricsserver/main.go:154 +0xdef

If I manually modify the deployment to reference the correct secret then all works correctly.
I can also set githubWebhookServer.secret.name in the chart values, but that will change the secret the webhook server is referencing as well.

Describe the expected behavior

Should reference the correct secret.
The metrics-server is unique in that it needs both the webhook secret and the github credentials, so probably needs another set of secret related values in the chart?

Probably a couple other improvements that could be made at the same time.
As I understand it the metrics-server requires the github client to be working, maybe better to throw an error and exit if the client can't be initialised at startup?

There's no doco for the metrics-server yet? Not that I could find anyway, so taken a bit of faffing around to even get this far!

Whole Controller Logs
N/A
Whole Runner Pod Logs
N/A
Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with charts/actions-runner-controller/templates/actionsmetrics.deployment.yaml at the referenced lines, then compare its credential references with the controller deployment and the chart values for secret settings. Render the Helm chart with actionsMetricsServer enabled and verify that the metrics server uses the GitHub credentials secret while webhook-secret references remain separate.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes
Domain
authentication, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.