lance-format / lance-format/lance
Allow users to specify whether they prefer speed or compression
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Also, once we have a better idea of the performance we should see if LZ4 is a "clear win". I'm worried it will be slower to use LZ4 in many cases and some customers would rather store a few more GB than deal with a slower reader. Maybe in the future we can introduce one simple config in the reader config (not field metadata) which specifies "prefer compression" or "prefer speed".
Originally posted by @westonpace in https://github.com/lancedb/lance/pull/4177#discussion_r2197858482
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the reader configuration and the code that selects or evaluates compression, then compare the performance implications of LZ4 for readers. Define what “prefer compression” and “prefer speed” should mean and verify that the choice belongs in reader configuration rather than field metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100