quickwit-oss / quickwit-oss/quickwit
Services using gRPC will break when hitting the max gRPC message size limit
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
gRPC message default size limit (which I think was introduced in tonic 0.9) is set to 4MB.
With a lot of splits or indexes, a gRPC call to the metastore will return an error. For now, we saw a few examples happening on a quickwit cluster:
- a search request on tens thousand splits will hit the limit. The search request errors after calling
list_splits(the merge planner too was concerned but we fixed the request to decrease the number of requested splits, see #3585) - with an index with a lot of partitions,
list_indexes_metadataswill error. This call is particularly critical.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the metastore gRPC definitions and the implementations of list_splits and list_indexes_metadatas, then review how the tonic message-size limit is configured. Reproduce the failure with many splits or partitions and determine the intended handling for responses exceeding 4 MB. Done means both calls work for the reported large-index cases without hitting the limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100