apache / apache/fluss

[rust] Support writing records to multiple tables

Open
#3,961 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

The Java client supports writing records to multiple tables through `MultiTableWriter` (introduced with #3139 / #3140), but the Rust client still requires callers to resolve each table and manage a separate `AppendWriter` or `UpsertWriter` themselves.

This is a recurring requirement for database CDC and DTS pipelines: one source stream contains inserts, updates, and deletes for many source tables. Without a Rust multi-table writer, every connector has to duplicate table metadata lookup, schema-aware writer caching, operation validation, and connection-wide flushing.

A concrete downstream use case is a Rust CDC engine that needs to route snapshot and change records from many source tables into Fluss while preserving each record's schema id.

### Solution

Add a Rust API equivalent in scope to the Java multi-table writer:

- A write record carrying `TablePath`, operation (`APPEND`, `UPSERT`, or `DELETE`), row, and schema id.
- A writer that resolves and caches per-table write state.
- `APPEND` for log tables and `UPSERT` / `DELETE` for primary-key tables, with explicit validation for mismatched operations.
- A writer-level `flush` that acknowledges pending writes for all target tables.
- Tests covering multiple log and primary-key tables, delete operations, invalid table/operation combinations, schema ids, and flush behavior.

The implementation should reuse the existing Rust append/upsert encoding and routing code rather than add a new wire protocol.

### Anything else?

Java reference: `fluss-client/src/main/java/org/apache/fluss/client/table/writer/MultiTableWriter.java`.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Java reference in fluss-client/src/main/java/org/apache/fluss/client/table/writer/MultiTableWriter.java, then trace the existing Rust append/upsert encoding and routing code. Add tests for multiple log and primary-key tables, deletes, invalid operations, schema ids, and writer-level flush acknowledgements; done means these cases pass while reusing the existing protocol.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.