lance-format / lance-format/lance

`write_dataset` with multiple streams in parallel

Open
#2,771 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Problem

It's often hard trying to saturate IO throughput of object stores with a single read stream (which usually implies a single file is sequentially read from or written to)

Proposed Solution

Change Dataset::write from

pub async fn write(
    batches: impl RecordBatchReader + Send + 'static,
    uri: &str,
    params: Option<WriteParams>,
) -> Result<Self>

to

pub async fn write(
    batches: impl RecordBatchReader + Send + 'static,
    uri: &str,
    params: Option<WriteParams>,
) -> Result<Self> {
    write_streams([batches], uri, params).await
}

pub async fn write_streams(
    streams: [impl RecordBatchReader + Send + 'static],
    uri: &str,
    params: Option<WriteParams>,
) -> Result<Self>

Each of of the stream are expected to generate separate fragments. e.g. Say we have 2 streams and each stream contains 1001 rows. If we set max_rows_per_file = 1000, we should get 4 fragments of sizes -- [1000, 1, 1000, 1].

This design allows us to parallelize the writes in rust and avoids dealing with painful python threading.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Dataset::write and the proposed WriteParams usage, then trace how a single RecordBatchReader becomes output fragments. Add the write_streams API while preserving write as the single-stream path, and verify that two 1001-row streams with max_rows_per_file set to 1000 produce fragments sized [1000, 1, 1000, 1] in parallel.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.