lance-format / lance-format/lance
`write_dataset` with multiple streams in parallel
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Problem
It's often hard trying to saturate IO throughput of object stores with a single read stream (which usually implies a single file is sequentially read from or written to)
Proposed Solution
Change Dataset::write from
pub async fn write(
batches: impl RecordBatchReader + Send + 'static,
uri: &str,
params: Option<WriteParams>,
) -> Result<Self>
to
pub async fn write(
batches: impl RecordBatchReader + Send + 'static,
uri: &str,
params: Option<WriteParams>,
) -> Result<Self> {
write_streams([batches], uri, params).await
}
pub async fn write_streams(
streams: [impl RecordBatchReader + Send + 'static],
uri: &str,
params: Option<WriteParams>,
) -> Result<Self>
Each of of the stream are expected to generate separate fragments. e.g. Say we have 2 streams and each stream contains 1001 rows. If we set max_rows_per_file = 1000, we should get 4 fragments of sizes -- [1000, 1, 1000, 1].
This design allows us to parallelize the writes in rust and avoids dealing with painful python threading.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Dataset::write and the proposed WriteParams usage, then trace how a single RecordBatchReader becomes output fragments. Add the write_streams API while preserving write as the single-stream path, and verify that two 1001-row streams with max_rows_per_file set to 1000 produce fragments sized [1000, 1, 1000, 1] in parallel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100