apache / apache/iggy

connectors: Delta Lake Sink - allow concurrent writes across partitions

Open
#3,839 2 comments 0 reactions 1 assignee Claimed by @mmmmxa View on GitHub
connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

The `delta_sink` connector serializes all writes through a single shared lock for which all partitions compete. `DeltaSink` holds one `Mutex>` wrapping the Delta table handle, writer, and coercion tree. `consume()` acquires that lock and holds it across the actual write-to-object-store and commit-to-Delta-log calls (`writer.write()` and `writer.flush_and_commit()`). This limits the throughput of the sink, especially if the object store responds slowly.

Relevant code: `core/connectors/sinks/delta_sink/src/sink.rs::consume` (see the `TODO` comment above the lock acquisition).
Ref discussion: https://github.com/apache/iggy/pull/2889/#discussion_r2936719763

Another TODO in this algorithm is the implementation of retries for the parquet buffer writes. Since they are technically moving into a different place with this code, the retries have to be implemented here too.

### Affected area / component

Connectors

### Proposed solution

_This is still very much TBD, I am currently researching what could be the best solution. Once I am done with research, I will write my idea down as final and call up the contributors for a discussion._

Introduce a shared map keyed by partition ID where each partition's `consume()` call writes into its own buffer independently and concurrently. A separate piece of coordination logic could then periodically (time-based and/or write-count-based trigger — exact policy still TBD) combine buffers and flush/commit them to the Delta table. `consume()` would still only return once its partition's data has actually been durably committed — buffering into the shared map does not mean early-return/fire-and-forget; the caller blocks until its contribution is confirmed flushed, to preserve delivery guarantees.

### Alternatives considered

Currently researching alternatives.

### Contribution

- [x] I'm willing to submit a pull request to implement this feature

### Good first issue

- [ ] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.