Kubeclient does not support certificate authorities with intermediate certificate

The issue

We tried to run a Ruby application (using the kubeclient gem) on a kubernetes cluster which uses a custom CA (the cluster's CA itself is signed by another custom CA, hence the need for intermediate certificates).

Kubeclient (initialized from the KUBECONFIG file) fails with SSL verify errors.

How to reproduce

First, we create a KUBECONFIG file in the container by executing the following script:

write_client_kubeconfig() {
  KUBECTL=${1:?please provide the KUBECTL environment variable}

  # only needed for writing a kubeconfig:
  master_url=${MASTER_URL:-https://kubernetes.default.svc.cluster.local:443}
  master_ca=${MASTER_CA:-/var/run/secrets/kubernetes.io/serviceaccount/ca.crt}
  token_file=${TOKEN_FILE:-/var/run/secrets/kubernetes.io/serviceaccount/token}

  # set up configuration for openshift client
  if [ -n "${WRITE_KUBECONFIG:-''}" ]; then
      # craft a kubeconfig, usually at $KUBECONFIG location
      ${KUBECTL} config set-cluster master \
            --certificate-authority="${master_ca}" \
            --server="${master_url}"
      ${KUBECTL} config set-credentials account \
            --token="$(cat ${token_file})"
      ${KUBECTL} config set-context current \
            --cluster=master \
            --user=account \
            --namespace="${infra_project}"
      ${KUBECTL} config use-context current
  fi
}

write_client_kubeconfig kubectl

Then we try listing services cluster-wide using the kubectl binary:

kubectl get services --all-namespaces

This should work, assuming that the current service account is allowed to list services cluster-wide.

Finally, we try listing the same services in Ruby using kubeclient:

mkdir -p /tmp/test
cd /tmp/test

cat <<EOF | tee Gemfile
source 'https://rubygems.org'
gem 'kubeclient', '~> 4.8'
EOF

bundle install --path .bundle

cat <<EOF | tee test.rb
require 'kubeclient'

config = Kubeclient::Config.read(ENV.fetch('KUBECONFIG'))
context = config.context
ssl_options = context.ssl_options
auth_options = context.auth_options

client = Kubeclient::Client.new(
    context.api_endpoint, 'v1',
    ssl_options: ssl_options, auth_options: auth_options
)

services_names = client.get_services.map { |svc| svc.metadata.name }
puts services_names.inspect
EOF

bundle exec ruby test.rb

On clusters where the kubernetes CA has been signed by an intermediate CA, kubeclient fails to verify the kubernetes API certificate, even if the cacert in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt contains the intermediate certificates.

We see the following stacktrace:

 Kubeclient::HttpError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get issuer certificate)
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:130:in `rescue in handle_exception'
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:120:in `handle_exception'
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:567:in `fetch_entities'
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:554:in `load_entities'
/opt/deploy/.bundle/ruby/2.6.0/gems/kubeclient-4.6.0/lib/kubeclient/common.rb:134:in `discover'

The explanation

The reason of this behavior is the use of OpenSSL::X509::Store#add_cert in https://github.com/abonas/kubeclient/blob/v4.9.1/lib/kubeclient/config.rb#L58 .

Per the documentation:

add_cert(cert)
Adds the OpenSSL::X509::Certificate cert to the certificate store.

If we had used the add_file method instead of add_cert every certificate included in the cacert file would have been loaded.

Documentation of add_file:

add_file(file) → self
Adds the certificates in file to the certificate store. file is the path to the file, and the file contains one or more certificates in PEM format concatenated together.

I will submit a PR next

Start with lib/kubeclient/config.rb around the OpenSSL::X509::Store setup at line 58, then compare the Ruby OpenSSL store methods referenced in the issue. Reproduce with the provided KUBECONFIG and test.rb against a cluster using an intermediate CA; done means the Ruby client can list services when the CA file contains multiple certificates.