kragniz / kragniz/python-etcd3
"ConnectionFailedError: etcd connection failed" due to "InactiveRpcError of RPC"
- Dominant language
- Python
- Stars
- 450
- Forks
- 194
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
Suddenly we are facing connection problems between python-etcd3 client and etcd server. Unfortunately error occurs occasionally while prior and subsequent calls work fine. After short analysis the only difference between a stable version of our python app and current one was urllib3 version:
stable version= python-etcd3:0.12.0 / **urllib3:1.25.11**
current problematic version= python-etcd3:0.12.0 / **urllib3:1.26.3**
Ccode snippet:
``` python
import etcd3
etcd = etcd3.client(host=etcd_host, port=etcd_port)
keyValueGenTuples = etcd.get_all()
keyValueTuples = [(kvgt[1].key, kvgt[0]) for kvgt in keyValueGenTuples] # Error occurs while iterating generator
```
Log:
```
File "/usr/local/lib/python3.8/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/etcd3/client.py", line 368, in get_all_response
return self.kvstub.Range(
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 923, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "{"created":"@1612710124.957218504","description":"Error received from peer ipv4:192.168.1.13:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Socket closed","grpc_status":14}"
```
etcd server itself says nothing! etcd server and python app are running on the same host, therefore network failures should be considered to be very unlikely.
So we gonna downgrade urllib3. We will let you know about the outcome.
Contributor guide
Research direction
Start in etcd3/client.py, especially get_all_response and the Range call shown in the traceback. Reproduce the intermittent failure while comparing urllib3 1.25.11 and 1.26.3, then establish whether the dependency change causes the Socket closed error; done requires a confirmed cause and a project-level resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python
- Domain
- api, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100