lance-format / lance-format/lance

Compressing binary data results in a very low compression ratio.

Open
#5,883 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

二进制文件(png、mp4)都在几百k,甚至几M,使用miniblock的话,基本上一个值就是一个块,使用full-zip的话,单值压缩,而且我们的场景是二进制数组,数组内元素也不怎么重复,导致压缩比比较低

Binary files (PNG, MP4) are typically several hundred kilobytes or even several megabytes in size. Using miniblock compression, essentially each value becomes a separate block. With full-zip compression, each value is compressed individually, and since our scenario involves binary arrays with few repeating elements, the compression ratio is quite low.

Image 这个是字段的schema定义 This is the schema definition for the field. Image 这个是压缩比 辛苦看看是不是哪里用的不对呢,或者少什么配置参数 This is the compression ratio. Please check carefully to see if anything is being used incorrectly or if any configuration parameters are missing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files, tests, or entry points are named. Start by reproducing the reported compression behavior for PNG and MP4 binary arrays with the shown schema, comparing miniblock and full-zip settings; determine whether the ratio is expected or whether a configuration or implementation issue is involved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.