kubernetes-client / kubernetes-client/python

Kubernetes client exception with connection refused error

Open
#2,041 1 comment 0 reactions 0 assignees View on GitHub
help wanted help-needed kind/bug kind/support
Dominant language
Python
Stars
7.7k
Forks
3.5k
Avg merge
1d 14h
Merged PRs (30d)
18

Description

Our tests had been running fine till we started running into the following issue where the kubernetes client websocket call will terminate with an exception:

channel = stream(api.connect_get_namespaced_pod_exec,
/usr/lib/python3.8/site-packages/kubernetes/stream/stream.py:35: in _websocket_request
return api_method(*args, **kwargs)
/usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:994: in connect_get_namespaced_pod_exec
return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501
/usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:1101: in connect_get_namespaced_pod_exec_with_http_info
return self.api_client.call_api(
/usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
return self.__call_api(resource_path, method,
/usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
response_data = self.request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
configuration =
_method = 'GET'
url = 'wss://api.ndal-ocp2.ve.pd.f5net.com:6443/api/v1/namespaces/spk-test-automation/pods/ans-client-ci6492742-127/exec?com...command=2001%3A0db8%3A0%3A127%3A%3A100&command=-t&command=10&command=-P8&stderr=True&stdin=False&stdout=True&tty=False'
kwargs = {'_preload_content': False, '_request_timeout': None, 'body': None, 'headers': {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256~bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'}, ...}
headers = {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256~bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'}
_request_timeout = None, _preload_content = False, capture_all = True
def websocket_call(configuration, _method, url, **kwargs):
"""An internal function to be called in api-client when a websocket
connection is required. method, url, and kwargs are the parameters of
apiClient.request method."""

url = get_websocket_url(url, kwargs.get("query_params"))
headers = kwargs.get("headers")
_request_timeout = kwargs.get("_request_timeout", 60)
_preload_content = kwargs.get("_preload_content", True)
capture_all = kwargs.get("capture_all", True)

try:
client = WSClient(configuration, url, headers, capture_all)
if not _preload_content:
return client
client.run_forever(timeout=_request_timeout)
return WSResponse('%s' % ''.join(client.read_all()))
except (Exception, KeyboardInterrupt, SystemExit) as e:
> raise ApiException(status=0, reason=str(e))
E kubernetes.client.exceptions.ApiException: (0)
E Reason: [Errno 111] Connection refused
/usr/lib/python3.8/site-packages/kubernetes/stream/ws_client.py:525: ApiException

These exceptions happen at random (not for any specific or every test) and across different kubernetes environment/clusters. They started happening somewhere around early Feb, 2023. (not sure if its related to any fixes/changes).

Below is what we use in our code:
`def k8s_exec(command, pod, namespace, api):
channel = stream(api.connect_get_namespaced_pod_exec,
pod,
namespace,
command=command,
stderr=True, stdin=False,
stdout=True, tty=False,
_preload_content=False)
channel.run_forever(timeout=60)
err = yaml.safe_load(channel.read_channel(ERROR_CHANNEL))
if err is None:
raise AssertionError(f'The command "{command}" was timeouted')
rc = 0 if err['status'] == 'Success' else 1
stdout = channel.read_stdout()
stderr = channel.read_stderr()

return rc, stdout, stderr`

Not sure if this is similar to: [1841](https://github.com/kubernetes-client/python/issues/1841)

**What you expected to happen**:
Connection is retained.

**Environment**:
- Kubernetes version (`kubectl version`):
Client Version: version.Info{Major:"1", Minor:"20"
Server Version: version.Info{Major:"1", Minor:"23"
- OS (e.g., MacOS 10.13.6): Linux
- Python version (`python --version`): 2.7.5

Contributor guide

Open the contributing guide

Research direction

Start with the k8s_exec function and the websocket_call path in kubernetes/stream/ws_client.py, especially the failure reported at line 525. Reproduce the intermittent connection-refused error across the stated Kubernetes environments and compare it with issue 1841. Done means identifying and addressing the cause so websocket connections remain available during pod exec calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.