kragniz / kragniz/python-etcd3

Here's a workaround for exceeding gRPC message size with get_prefix().

Open
#1,228 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
450
Forks
194
PR merge metrics
No merged PRs in 30d

Description

When calling get_prefix() against an etcd v3 source with >175,000 keys, I got an error like this:

File ".../lib/python2.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Received message larger than max (10666130 vs. 4194304)"
debug_error_string = "{"created":"@1594399189.035786331","description":"Received message larger than max (10666130 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":189,"grpc_status":8}"

Building on [gjcarneiro's work](https://www.gitmemory.com/issue/etcd-io/etcd/11001/519213542), here's a fleshed out function with a working example ([list-keys.txt](https://github.com/kragniz/python-etcd3/files/4905745/list-keys.txt)). It walks through keys in chunks and adjusts that chunk size down until it finds one that doesn't error out.

Attached is the code. Is there a better place to put it?

Contributor guide

Open the contributing guide

Research direction

Start with the get_prefix() entry point and review the attached list-keys.txt workaround alongside the reported gRPC message-size error. Determine whether this chunked behavior belongs in the client API and what placement is appropriate; the issue is done when that location and the handling of oversized responses are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python
Domain
api, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.