lance-format / lance-format/lance
Compressing binary data results in a very low compression ratio.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
二进制文件(png、mp4)都在几百k,甚至几M,使用miniblock的话,基本上一个值就是一个块,使用full-zip的话,单值压缩,而且我们的场景是二进制数组,数组内元素也不怎么重复,导致压缩比比较低
Binary files (PNG, MP4) are typically several hundred kilobytes or even several megabytes in size. Using miniblock compression, essentially each value becomes a separate block. With full-zip compression, each value is compressed individually, and since our scenario involves binary arrays with few repeating elements, the compression ratio is quite low.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files, tests, or entry points are named. Start by reproducing the reported compression behavior for PNG and MP4 binary arrays with the shown schema, comparing miniblock and full-zip settings; determine whether the ratio is expected or whether a configuration or implementation issue is involved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100