kubernetes-client / kubernetes-client/python

IDP certificate issue

Open
#2,415 13 comments 2 reactions 1 assignee Claimed by @ealeonraz View on GitHub
help wanted kind/bug
Dominant language
Python
Stars
7.7k
Forks
3.5k
Avg merge
1d 14h
Merged PRs (30d)
18

Description

This might be basically the same as #2160, or maybe it's just related. I'm not sure, but since this issue is closed I thought I should open a new one.

We're using TKGI to create and manage Kubernetes clusters. TKGI is also the IDP (OIDC) for the clusters. When I run `tkgi get-credentials `, a `~/.kube/config` is created which contains the necessary information.

When the access token expires, kubectl is perfectly able to get a new one with the info from the config. But this Python lib isn't. I think the problem is that TKGI only writes the certificat and not the complete chain into `idp-certificate-authority-data`.

Please let me stress again that kubectl does **not** have a problem with that!

**What happened (please include outputs or screenshots)**:
```
$ python
Python 3.12.8 (main, Dec 9 2024, 15:25:01) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from kubernetes import client, config
>>> config.load_config()
Traceback (most recent call last):
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
self._validate_conn(conn)
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
conn.connect()
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connection.py", line 741, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/util/ssl_.py", line 460, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/util/ssl_.py", line 504, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/ssl.py", line 455, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.12/ssl.py", line 1319, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1000)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "", line 1, in
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/config/__init__.py", line 42, in load_config
load_kube_config(**kwargs)
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/config/kube_config.py", line 826, in load_kube_config
loader.load_and_set(config)
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/config/kube_config.py", line 589, in load_and_set
self._load_authentication()
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/config/kube_config.py", line 288, in _load_authentication
if self._load_auth_provider_token():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/config/kube_config.py", line 307, in _load_auth_provider_token
return self._load_oid_token(provider)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/config/kube_config.py", line 413, in _load_oid_token
self._refresh_oidc(provider)
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/config/kube_config.py", line 450, in _refresh_oidc
response = client.request(
^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/client/rest.py", line 244, in GET
return self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/kubernetes/client/rest.py", line 217, in request
r = self.pool_manager.request(method, url,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/_request_methods.py", line 135, in request
return self.request_encode_url(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
return self.urlopen(method, url, **extra_kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/poolmanager.py", line 443, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/data/venv-ansible-vmware/lib64/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='tkgi-playground.it.nrw.de', port=8443): Max retries exceeded with url: /oauth/token/.well-known/openid-configuration (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1000)')))
>>>
```

**What you expected to happen**:
```
$ python
Python 3.12.8 (main, Dec 9 2024, 15:25:01) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from kubernetes import client, config
>>> config.load_config()
>>>
```

**How to reproduce it (as minimally and precisely as possible)**:
Test with `idp-certificate-authority-data` containing only the certificat, not the complete chain.

**Anything else we need to know?**:

When I manually patch `idp-certificate-authority-data` in `~/.kube/config` to contain the complete certificat chain everything works. I think the problem is somewhere around here:

https://github.com/kubernetes-client/python/blob/3e6cc5871997cd52fd8f4c8324894508f3a75136/kubernetes/base/config/kube_config.py#L422-L454

When connecting to TKGI, there's the complete certificat chain. Neither kubectl nor curl nor anything else has a problem there. But this lib has, because it trusts only the certificat from `idp-certificate-authority-data`, which lacks the intermediate and root CA certificats.

**Environment**:
- Kubernetes version (`kubectl version`):
```
Client Version: v1.30.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.5+vmware.1
```
- OS (e.g., MacOS 10.13.6): Red Hat Enterprise Linux release 8.10 (Ootpa)
- Python version (`python --version`): 3.12.8
- Python client version (`pip list | grep kubernetes`): 32.0.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.