quickwit-oss / quickwit-oss/quickwit
Rest payload limits vs gRPC payload limit impedance.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
The largest accepted payload limit in rest is configurable.
If the node receiving the request is a searcher, the payload is forwarded in gRPC to an indexer.
The payload is then limited to 10MiB.
I increased this value in #4227 to at least match the default value in our rest client, but we probably should do more.
- have a very high limit on the gRPC side and use that as a cap for the rest limit?
- chop payload in the rest client?
Also we should probably see if we can preconfigure a nginx ingress in helm to route ingest request to indexers, and search requests to searchers.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how configurable REST payload limits are forwarded from searchers to indexers over gRPC, then review the Helm ingress configuration. Decide how the REST and gRPC caps should relate and whether ingest and search requests can be routed separately. Done means the chosen limit behavior and ingress routing are implemented and verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, helm, nginx, rust
- Domain
- cloud, distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100