lance-format / lance-format/lance

Allow batch size to be specified in bytes in file reader for 2.0

Open
#4,369 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Currently the file reader requires a batch size to be specified in rows. This is difficult because the ideal batch size is usually expressed in bytes (e.g. "fits in cpu cache" or "less than 20MB") and users have to calibrate the batch size to the shape of the data they are loading (and they never do this).

It would be much better to allow a target batch size to be expressed in bytes. Note: this won't be all that trivial. We will need to enhance the structural decoders to support this operation. It isn't too hard to solve for one column but when reading multiple columns it can be difficult to know where exactly to break the batch.

I think a simple guess-and-check algorithm can probably strike a good balance (and ensures we always have a power of 2 batch size which has its own advantages):

  • Is 8Ki rows close? (nothing magical about 8Ki but it should be a power of 2 and could be dependent on the batch size bytes)
  • If no, then double / halve as appropriate
  • Repeat until we get close (or get to min_batch_size rows or max_batch_size rows)
  • Remember the value used and start from there on the next call

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the file reader and structural decoder entry points to understand how row-based batches are formed and sized. Investigate how multi-column batches could be measured in bytes while preserving power-of-two sizing and the stated minimum and maximum row limits. Done means callers can request a byte target and batches converge toward it, with the chosen size reused on the next call.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.