openshift / openshift/hypershift
Bootstrap kubeconfig does not include named serving certificate CAs
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 541
- Forks
- 567
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 136
Description
Description
When a HostedCluster uses Route-based service publishing (servicePublishingStrategy.type: Route)
and the KAS serving certificate is signed by a non-HyperShift CA (e.g. a corporate PKI configured
via spec.configuration.apiServer.servingCerts.namedCertificates), worker nodes fail to connect to
the API server with:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Root Cause
The worker node's kubelet kubeconfig (/var/lib/kubelet/kubeconfig) embeds only the HyperShift
root-ca as its certificate-authority-data. On Route-based clusters the local
kube-apiserver-proxy (HAProxy at 172.20.0.1:6443) does TCP passthrough to the KAS Route, so
the kubelet must verify the KAS TLS certificate directly.
spec.additionalTrustBundle correctly flows into the MCO user CA bundle on the node
(/etc/pki/ca-trust/source/anchors/), but the kubelet does not use the system trust store for API
server verification; it uses only the CA embedded in its kubeconfig.
The code path that builds the bootstrap kubeconfig (adaptBootstrapKubeconfigSecret in
control-plane-operator/.../v2/kas/kubeconfig.go) uses GenerateKubeConfig, which only fetches
root-ca. Compare with adaptCustomAdminKubeconfigSecret, which already calls
combineRootCAWithServingCerts to include named certificate CAs. The bootstrap kubeconfig does
not receive the same treatment.
Expected Behaviour
The bootstrap kubeconfig's certificate-authority-data should include:
- The HyperShift
root-ca - CAs from
spec.configuration.apiServer.servingCerts.namedCertificates(consistent with the admin kubeconfig)
This would allow worker kubelets to verify KAS serving certificates signed by external CAs when
using Route-based service publishing.
Steps to Reproduce
- Create a HostedCluster with
servicePublishingStrategy.type: Route - Configure
spec.configuration.apiServer.servingCerts.namedCertificateswith a certificate
signed by a non-HyperShift CA - Provision a worker node
- Observe kubelet logs:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Workaround
A MachineConfig can be applied via the NodePool to inject a combined CA bundle (HyperShift root-ca +
external CA) into the kubelet kubeconfigs at boot time using a systemd oneshot service ordered
before kubelet.service. This is fragile and requires manual maintenance when CAs rotate.
Related
- Previous upstream issue: #3985 (closed by bot inactivity, no code fix)
- Jira: OCPBUGS-41853 (resolved with docs-only fix in 4.19.0)
- KCS: 7111089
- Fix PR: #8311
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in control-plane-operator/.../v2/kas/kubeconfig.go at adaptBootstrapKubeconfigSecret and compare it with adaptCustomAdminKubeconfigSecret, especially combineRootCAWithServingCerts. Check the existing kubeconfig-related tests, then verify that bootstrap kubeconfig certificate-authority-data covers both root-ca and named serving certificate CAs for Route-based publishing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100