actions / actions/runner-container-hooks

Runners fail with ERR_TLS_CERT_ALTNAME_INVALID when preparing job on an IPv6 EKS cluster.

Open
#245 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
153
Forks
112
Avg merge
6m
Merged PRs (30d)
1

Description

Checks
Controller Version

0.12.1

Deployment Method

Helm

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Deploy ARC and a RSS to an IPv6-enabled cluster. Runner is 'kubernetes' mode.
2. Create a simple dispatch workflow that `runs-on: myrunner`
3. Dispatch.
4. Runner crashes during 'Initialize Containers'.
Describe the bug

When trying to migrate our runners to an IPv6 EKS cluster, we find that the runners consistently crash in the 'Initialize Containers' step.

Error [ERR_TLS_CERT_ALTNAME_INVALID]: Hostname/IP does not match certificate's altnames: Host: fd26. is not in the cert's altnames: DNS:c699ee59bb9e133834eae210f228abc6.yl4.eks-cluster.us-east-1.api.aws, DNS:ip-172-16-172-216.ec2.internal, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:FD26:11D8:2382:0:0:0:0:1, IP Address:2600:1F18:427C:8111:0:0:0:F22, IP Address:172.16.172.216

This is a suspicious message, especially Host: fd26. is not. fd26 is the first part of my cluster's "Service IPv6 Range": fd26:11d8:2382::/108. In fact, KUBERNETES_SERVICE_HOST=fd26:11d8:2382::1. And the DNS result for kubernetes.default is

$ getent hosts kubernetes.default
fd26:11d8:2382::1 kubernetes.default.svc.cluster.local

This Host: fd26. is an outcome I would theorize coming from improper handling of host addresses; attempting to split an assumed address+port string on :, then taking the first part as the address.

Describe the expected behavior

I expect that the runner should work on IPv6 without modification or override.

When migrating workloads to IPv6, I often encounter improper address handling in every language from Ruby, Javascript, Python and Go. Implementers often assume they can string-build a URI from component strings, which falls apart at the edge cases. All of these migrations were solved by correcting implementation to use proper URI-handling standard library functions, or by formatting the address using the bracketed notation: [feeb:beef::1]:8080, which overcomes some issues in third-party libraries and tools.

I attempted to find the root cause of the issue in this repository, @actions/runner-container-hooks and @actions/runner but was unable to pinpoint it.

This effectively blocks me (and apparently everyone) from deploying ARC to IPv6-only clusters.

Controller Logs
https://gist.github.com/carl-reverb/5666aa77be92f57e16320d89c0f2c2db
Runner Pod Logs
https://gist.github.com/carl-reverb/40ac08236942c9bb2aa8330e78fb5c7f

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the IPv6 EKS deployment described in the issue, then read the controller and runner pod logs linked in the report while tracing the Initialize Containers step. The issue names no source file or test; done means the runner completes initialization on an IPv6-only cluster without a TLS hostname error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes, typescript
Domain
cloud, devops, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.