kubernetes-client / kubernetes-client/python
Kubernetes client exception with connection refused error
- Dominant language
- Python
- Stars
- 7.7k
- Forks
- 3.5k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 18
Description
Our tests had been running fine till we started running into the following issue where the kubernetes client websocket call will terminate with an exception:
channel = stream(api.connect_get_namespaced_pod_exec,
/usr/lib/python3.8/site-packages/kubernetes/stream/stream.py:35: in _websocket_request
return api_method(*args, **kwargs)
/usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:994: in connect_get_namespaced_pod_exec
return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501
/usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:1101: in connect_get_namespaced_pod_exec_with_http_info
return self.api_client.call_api(
/usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
return self.__call_api(resource_path, method,
/usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
response_data = self.request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
configuration =
_method = 'GET'
url = 'wss://api.ndal-ocp2.ve.pd.f5net.com:6443/api/v1/namespaces/spk-test-automation/pods/ans-client-ci6492742-127/exec?com...command=2001%3A0db8%3A0%3A127%3A%3A100&command=-t&command=10&command=-P8&stderr=True&stdin=False&stdout=True&tty=False'
kwargs = {'_preload_content': False, '_request_timeout': None, 'body': None, 'headers': {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256~bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'}, ...}
headers = {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256~bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'}
_request_timeout = None, _preload_content = False, capture_all = True
def websocket_call(configuration, _method, url, **kwargs):
"""An internal function to be called in api-client when a websocket
connection is required. method, url, and kwargs are the parameters of
apiClient.request method."""
url = get_websocket_url(url, kwargs.get("query_params"))
headers = kwargs.get("headers")
_request_timeout = kwargs.get("_request_timeout", 60)
_preload_content = kwargs.get("_preload_content", True)
capture_all = kwargs.get("capture_all", True)
try:
client = WSClient(configuration, url, headers, capture_all)
if not _preload_content:
return client
client.run_forever(timeout=_request_timeout)
return WSResponse('%s' % ''.join(client.read_all()))
except (Exception, KeyboardInterrupt, SystemExit) as e:
> raise ApiException(status=0, reason=str(e))
E kubernetes.client.exceptions.ApiException: (0)
E Reason: [Errno 111] Connection refused
/usr/lib/python3.8/site-packages/kubernetes/stream/ws_client.py:525: ApiException
These exceptions happen at random (not for any specific or every test) and across different kubernetes environment/clusters. They started happening somewhere around early Feb, 2023. (not sure if its related to any fixes/changes).
Below is what we use in our code:
`def k8s_exec(command, pod, namespace, api):
channel = stream(api.connect_get_namespaced_pod_exec,
pod,
namespace,
command=command,
stderr=True, stdin=False,
stdout=True, tty=False,
_preload_content=False)
channel.run_forever(timeout=60)
err = yaml.safe_load(channel.read_channel(ERROR_CHANNEL))
if err is None:
raise AssertionError(f'The command "{command}" was timeouted')
rc = 0 if err['status'] == 'Success' else 1
stdout = channel.read_stdout()
stderr = channel.read_stderr()
return rc, stdout, stderr`
Not sure if this is similar to: [1841](https://github.com/kubernetes-client/python/issues/1841)
**What you expected to happen**:
Connection is retained.
**Environment**:
- Kubernetes version (`kubectl version`):
Client Version: version.Info{Major:"1", Minor:"20"
Server Version: version.Info{Major:"1", Minor:"23"
- OS (e.g., MacOS 10.13.6): Linux
- Python version (`python --version`): 2.7.5
Contributor guide
Research direction
Start with the k8s_exec function and the websocket_call path in kubernetes/stream/ws_client.py, especially the failure reported at line 525. Reproduce the intermittent connection-refused error across the stated Kubernetes environments and compare it with issue 1841. Done means identifying and addressing the cause so websocket connections remain available during pod exec calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100