kragniz / kragniz/python-etcd3

"ConnectionFailedError: etcd connection failed" due to "InactiveRpcError of RPC"

Open
#1,467 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
450
Forks
194
PR merge metrics
No merged PRs in 30d

Description

Hello,

Suddenly we are facing connection problems between python-etcd3 client and etcd server. Unfortunately error occurs occasionally while prior and subsequent calls work fine. After short analysis the only difference between a stable version of our python app and current one was urllib3 version:

stable version= python-etcd3:0.12.0 / **urllib3:1.25.11**
current problematic version= python-etcd3:0.12.0 / **urllib3:1.26.3**

Ccode snippet:
``` python
import etcd3
etcd = etcd3.client(host=etcd_host, port=etcd_port)

keyValueGenTuples = etcd.get_all()
keyValueTuples = [(kvgt[1].key, kvgt[0]) for kvgt in keyValueGenTuples] # Error occurs while iterating generator
```

Log:
```
File "/usr/local/lib/python3.8/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/etcd3/client.py", line 368, in get_all_response
return self.kvstub.Range(
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 923, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "{"created":"@1612710124.957218504","description":"Error received from peer ipv4:192.168.1.13:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Socket closed","grpc_status":14}"

```
etcd server itself says nothing! etcd server and python app are running on the same host, therefore network failures should be considered to be very unlikely.

So we gonna downgrade urllib3. We will let you know about the outcome.

Contributor guide

Open the contributing guide

Research direction

Start in etcd3/client.py, especially get_all_response and the Range call shown in the traceback. Reproduce the intermittent failure while comparing urllib3 1.25.11 and 1.26.3, then establish whether the dependency change causes the Socket closed error; done requires a confirmed cause and a project-level resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python
Domain
api, backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.