lance-format / lance-format/lance
Filtering file: Parquet and Feather twice as fast
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
I look forward to moving some workload to Lance, so I ran a test for the filtering performance.
(for retrieving data by row number, Lance is way better than Parquet or Feather)
When filtering a dataset, I had this performance:
- Lance avg processing time 33 seconds
- Feather avg processing time 14 seconds
- Parquet avg processing time 19 seconds
Is it expected that the other formats are about twice as fast for filtering?
Here is the benchmark code on Google Colab:
https://colab.research.google.com/drive/1iG1MXJV-9hrqm4YcO_MzR0Qt67sMFfEj?usp=sharing
It's the benchmark 2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the benchmark code in the linked Google Colab and verify the reported filtering timings for Lance, Feather, and Parquet. Compare the benchmark setup and filtering behavior, then document whether the difference is expected or identify a reproducible performance problem.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100