quickwit-oss / quickwit-oss/quickwit

Services using gRPC will break when hitting the max gRPC message size limit

Open
#3,626 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

gRPC message default size limit (which I think was introduced in tonic 0.9) is set to 4MB.

With a lot of splits or indexes, a gRPC call to the metastore will return an error. For now, we saw a few examples happening on a quickwit cluster:

  • a search request on tens thousand splits will hit the limit. The search request errors after calling list_splits (the merge planner too was concerned but we fixed the request to decrease the number of requested splits, see #3585)
  • with an index with a lot of partitions, list_indexes_metadatas will error. This call is particularly critical.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the metastore gRPC definitions and the implementations of list_splits and list_indexes_metadatas, then review how the tonic message-size limit is configured. Reproduce the failure with many splits or partitions and determine the intended handling for responses exceeding 4 MB. Done means both calls work for the reported large-index cases without hitting the limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.