quickwit-oss / quickwit-oss/quickwit
Search requests with high `max_hits` (> ~100) causes fetch error with indexes stored on Cloudflare R2.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
Search requests with high max_hits (> ~100) causes fetch errors with indexes stored on Cloudflare R2.
This is likely because Quickwit goes over the Cloudflare R2 limit of 1000 GetObject requests per second: https://developers.cloudflare.com/r2/platform/limits/#account-plan-limits
As discussed on discord, it would be helpful if Quickwit propagates the Cloudflare R2 error message to confirm this theory.
Steps to reproduce (if applicable)
Steps to reproduce the behavior:
- Search request:
❯ curl http://127.0.0.1:7280/api/v1/reddit_comments_v05/search\?query\=frankfurt\&max_hits\=1000\&sort_by_field\=-created_utc
{
"InternalError": "Internal error: `Error when fetching docs for splits [\"01G7YXYNFFVCQBY8Y2B5078EWW\", \"01G7YXPQQM8DER0F8112WK5C2P\", \"01G7YY6GH3N259P137D7TF6PA7\", \"01G7YWQHPA2Q1PHK8B0BXS0ZJP\"]: searcher-doc-async\n\nCaused by:\n An IO error occurred: 'Failed to fetch slice 1448721986..1449072261 for object: s3://test-bucket/indexes/reddit_comments_v05/01G7YXPQQM8DER0F8112WK5C2P.split'.`."
}%
Configuration:
Please provide:
- Output of
quickwit --version
❯ RUST_LOG=quickwit=debug bin/quickwit --version
Quickwit 0.3.1nightly
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the curl search against indexes stored on Cloudflare R2 with a high max_hits value, using the configuration and Quickwit version shown in the issue. Trace the searcher's fetch path from the reported “Failed to fetch slice” error and ensure the underlying Cloudflare R2 message is propagated; done means the response identifies the storage error rather than only reporting an internal fetch failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100