lance-format / lance-format/lance
Optimize block size based on query type
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We should benchmark the latency we get from different block sizes as well as how much concurrency we can get from different object stores.
Because of request rate limits, there's likely to be a tradeoff between latency (better with smaller block sizes) and throughput (better with larger block sizes). Depending on the type of query, we may want to optimize for one or the other. For example, a KNN query will likely want to optimize for latency. Whereas a OLAP scan will want to optimize for throughput. We should be able to provide good automatic defaults based on the query plan. Of course, the user can always input their own settings as needed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.