openshift / openshift/hypershift

Bootstrap kubeconfig does not include named serving certificate CAs

Open
#8,310 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

lifecycle/rotten
Dominant language
Go
Stars
541
Forks
567
Avg merge
5d 13h
Merged PRs (30d)
136

Description

Description

When a HostedCluster uses Route-based service publishing (servicePublishingStrategy.type: Route)
and the KAS serving certificate is signed by a non-HyperShift CA (e.g. a corporate PKI configured
via spec.configuration.apiServer.servingCerts.namedCertificates), worker nodes fail to connect to
the API server with:

tls: failed to verify certificate: x509: certificate signed by unknown authority

Root Cause

The worker node's kubelet kubeconfig (/var/lib/kubelet/kubeconfig) embeds only the HyperShift
root-ca as its certificate-authority-data. On Route-based clusters the local
kube-apiserver-proxy (HAProxy at 172.20.0.1:6443) does TCP passthrough to the KAS Route, so
the kubelet must verify the KAS TLS certificate directly.

spec.additionalTrustBundle correctly flows into the MCO user CA bundle on the node
(/etc/pki/ca-trust/source/anchors/), but the kubelet does not use the system trust store for API
server verification; it uses only the CA embedded in its kubeconfig.

The code path that builds the bootstrap kubeconfig (adaptBootstrapKubeconfigSecret in
control-plane-operator/.../v2/kas/kubeconfig.go) uses GenerateKubeConfig, which only fetches
root-ca. Compare with adaptCustomAdminKubeconfigSecret, which already calls
combineRootCAWithServingCerts to include named certificate CAs. The bootstrap kubeconfig does
not receive the same treatment.

Expected Behaviour

The bootstrap kubeconfig's certificate-authority-data should include:

  1. The HyperShift root-ca
  2. CAs from spec.configuration.apiServer.servingCerts.namedCertificates (consistent with the admin kubeconfig)

This would allow worker kubelets to verify KAS serving certificates signed by external CAs when
using Route-based service publishing.

Steps to Reproduce

  1. Create a HostedCluster with servicePublishingStrategy.type: Route
  2. Configure spec.configuration.apiServer.servingCerts.namedCertificates with a certificate
    signed by a non-HyperShift CA
  3. Provision a worker node
  4. Observe kubelet logs: tls: failed to verify certificate: x509: certificate signed by unknown authority

Workaround

A MachineConfig can be applied via the NodePool to inject a combined CA bundle (HyperShift root-ca +
external CA) into the kubelet kubeconfigs at boot time using a systemd oneshot service ordered
before kubelet.service. This is fragile and requires manual maintenance when CAs rotate.

Related

  • Previous upstream issue: #3985 (closed by bot inactivity, no code fix)
  • Jira: OCPBUGS-41853 (resolved with docs-only fix in 4.19.0)
  • KCS: 7111089
  • Fix PR: #8311

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in control-plane-operator/.../v2/kas/kubeconfig.go at adaptBootstrapKubeconfigSecret and compare it with adaptCustomAdminKubeconfigSecret, especially combineRootCAWithServingCerts. Check the existing kubeconfig-related tests, then verify that bootstrap kubeconfig certificate-authority-data covers both root-ca and named serving certificate CAs for Route-based publishing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure, security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.