kragniz / kragniz/python-etcd3
Here's a workaround for exceeding gRPC message size with get_prefix().
- Dominant language
- Python
- Stars
- 450
- Forks
- 194
- PR merge metrics
- No merged PRs in 30d
Description
When calling get_prefix() against an etcd v3 source with >175,000 keys, I got an error like this:
File ".../lib/python2.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Received message larger than max (10666130 vs. 4194304)"
debug_error_string = "{"created":"@1594399189.035786331","description":"Received message larger than max (10666130 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":189,"grpc_status":8}"
Building on [gjcarneiro's work](https://www.gitmemory.com/issue/etcd-io/etcd/11001/519213542), here's a fleshed out function with a working example ([list-keys.txt](https://github.com/kragniz/python-etcd3/files/4905745/list-keys.txt)). It walks through keys in chunks and adjusts that chunk size down until it finds one that doesn't error out.
Attached is the code. Is there a better place to put it?
Contributor guide
Research direction
Start with the get_prefix() entry point and review the attached list-keys.txt workaround alongside the reported gRPC message-size error. Determine whether this chunked behavior belongs in the client API and what placement is appropriate; the issue is done when that location and the handling of oversized responses are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python
- Domain
- api, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100