lance-format / lance-format/lance

Optimize block size based on query type

Open
#1,094 1 comment 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

enhancement rust
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

We should benchmark the latency we get from different block sizes as well as how much concurrency we can get from different object stores.

Because of request rate limits, there's likely to be a tradeoff between latency (better with smaller block sizes) and throughput (better with larger block sizes). Depending on the type of query, we may want to optimize for one or the other. For example, a KNN query will likely want to optimize for latency. Whereas a OLAP scan will want to optimize for throughput. We should be able to provide good automatic defaults based on the query plan. Of course, the user can always input their own settings as needed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.