lance-format / lance-format/lance
Avoid extra tokio spawn by arranging decode and filter task on the same thread
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Hmm, I'm not entirely sure I agree but I don't want to go back and forth too much. We can merge this and revisit later (I still want to get rid of some of the I/O tasks) if you would like.
I think we will also want a more complex benchmark, we could use one of the more compute intensive TPC-H queries.
We will also need to add support for FilteredReadThreadingMode::MultiplePartitions in the Lance table provider.
The goal should be that one thread task does decoding and filtering. This way when we reach the filtering stage, the data is already in the CPU cache. If we put a spawn here then the decoding will happen on one thread task and the filtering on another. This means we will have to transfer the data between main memory.
Originally posted by @westonpace in https://github.com/lancedb/lance/pull/5165#discussion_r2527481887
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the decode and filter tasks and the Lance table provider. Review how task spawning currently separates decoding from filtering, then examine the proposed TPC-H benchmark and support for FilteredReadThreadingMode::MultiplePartitions. Done means decoding and filtering run on the same thread task and the additional threading mode is supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100