lance-format / lance-format/lance

bug: binary_quantization appends a spurious byte when the length is a multiple of 8

Open Beginner friendly
#9,028 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

binary_quantization packs the sign bits of a float slice into bytes and appends the tail byte unconditionally (rust/lance-index/src/vector/bq.rs:69):

let iter = data.chunks_exact(8);
iter.clone()
    .map(|c| { ... })
    .chain(once(0).map(move |_| {
        let mut bits: u8 = 0;
        iter.remainder().iter().enumerate().for_each(...);
        bits
    }))

When data.len() is a multiple of 8 the remainder is empty, so the chained byte is always zero and the output is floor(len / 8) + 1 bytes instead of ceil(len / 8).

The public entry point BinaryQuantization::transform flattens every row through this helper into one UInt8Array, so the per-row width a caller has to assume, ceil(dim / 8), disagrees with what it gets. For a dimension that is a multiple of 8, every row carries one extra zero byte.

Reproduced on 36bd4e27a: 16 values give [85, 85, 0] where [85, 85] is correct, and transform over 3 rows of dimension 8 returns 6 bytes instead of 3.

BinaryQuantization has no callers in this repository, so this is latent rather than a live failure, and RabitQuantizer is what the IVF_RQ path uses. Filing it because the type is public API (lance_index::vector::bq::BinaryQuantization) and the helper predates RaBitQ (#1988).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance-index/src/vector/bq.rs at the binary_quantization helper and trace how BinaryQuantization::transform flattens rows. Verify the 16-value example and the three-row dimension-8 case, then ensure dimensions divisible by 8 produce ceil(dim / 8) bytes with no extra tail byte.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.