quickwit-oss / quickwit-oss/quickwit
Performance falls rapidly as max_hits increases in size
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Was poking around trying to isolate some weird sorting behavior and noticed that query times increase rapidly as the size of max_hits goes up. This is running a build from main, single node with local file system storage.
Index info
1. General infos
===============================================================================
Index id: noaa-1m
Index uri: file:///Users/kstaken/projects/opensource/quickwit-test/qwdata/indexes/noaa-1m
Number of published splits: 2
Number of published documents: 1634467
Size of published splits: 84 MB
$ curl -sS 'http://127.0.0.1:7280/api/v1/noaa-1m/search?query=name:NO&max_hits=10' | jq .elapsed_time_micros
46021
$ curl -sS 'http://127.0.0.1:7280/api/v1/noaa-1m/search?query=name:NO&max_hits=100' | jq .elapsed_time_micros
68718
$ curl -sS 'http://127.0.0.1:7280/api/v1/noaa-1m/search?query=name:NO&max_hits=1000' | jq .elapsed_time_micros
252134
$ curl -sS 'http://127.0.0.1:7280/api/v1/noaa-1m/search?query=name:NO&max_hits=10000' | jq .elapsed_time_micros
2832835
$ curl -sS 'http://127.0.0.1:7280/api/v1/noaa-1m/search?query=name:NO&max_hits=100000' | jq .elapsed_time_micros
null
For the 100,000 record case the server panics as it's out of file descriptors.
2022-05-13T18:36:10.966Z ERROR fetch_docs: quickwit_search::fetch_docs: Error when fetching docs in splits. split_ids=["01G2Z5RC4880BTWNSKX3P6YM9N", "01G2Z4B7QHBRE31G3203BWVV6N"] error=searcher-doc-async
Caused by:
0: An IO error occurred: 'Too many open files (os error 24)'
1: Too many open files (os error 24)
2022-05-13T18:36:13.817Z ERROR fetch_docs: quickwit_search::fetch_docs: Error when fetching docs in splits. split_ids=["01G2Z5RC4880BTWNSKX3P6YM9N", "01G2Z4B7QHBRE31G3203BWVV6N"] error=searcher-doc-async
Caused by:
0: An IO error occurred: 'Too many open files (os error 24)'
1: Too many open files (os error 24)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the single-node local-file-system queries against the noaa-1m index using the shown curl commands and compare elapsed times as max_hits increases. Start from the search endpoint and the fetch_docs error context; done means large max_hits requests no longer cause rapid degradation or Too many open files failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100