[rust/python] Writing to datalake-enabled Iceberg tables fails: "KeyEncoder for Iceberg format is not yet implemented"
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Fluss version
0.9.1-incubating (server), pyfluss 0.1.0 (client). The gap is still present on current `main`.
### Please describe the bug 🐞
Any write to a primary-key table with `table.datalake.enabled = 'true'` and Iceberg as the
cluster's datalake format fails from the Python client (`pyfluss`, backed by `fluss-rust`)
with:
```
KeyEncoder for Iceberg format is not yet implemented
```
The same write against the same table succeeds with the Java client / Flink connector, so
the table itself is fine — the limitation is client-side.
Root cause: in `fluss-rust/crates/fluss/src/row/encode/mod.rs`, the key-encoder factory
implements Paimon and Lance but explicitly rejects Iceberg:
```rust
match data_lake_format {
Some(DataLakeFormat::Paimon) => {
Ok(Box::new(PaimonKeyEncoder::new(row_type, key_fields)?))
}
Some(DataLakeFormat::Lance) => Ok(Box::new(CompactedKeyEncoder::create_key_encoder(
row_type, key_fields,
)?)),
Some(DataLakeFormat::Iceberg) => Err(Error::UnsupportedOperation {
message: "KeyEncoder for Iceberg format is not yet implemented".to_string(),
}),
None => Ok(Box::new(CompactedKeyEncoder::create_key_encoder(
row_type, key_fields,
)?)),
}
```
The Java client encodes keys for Iceberg-enabled tables (bucket-id computation compatible
with Iceberg's bucket transform), so the Rust client needs the equivalent encoder to reach
parity.
### Impact
This blocks Python-based pipelines (e.g. CDC ingestion via pyfluss) from writing to any
datalake-enabled table on Iceberg clusters — the datalake feature and the Python client are
mutually exclusive today. Reported by a production user running Fluss 0.9.1-incubating with
pyfluss 0.1.0.
### Steps to reproduce
1. Run a Fluss cluster with `datalake.format: iceberg`.
2. Create a primary-key table with `'table.datalake.enabled' = 'true'`.
3. From Python: connect with pyfluss 0.1.0 and attempt any upsert into the table.
4. The write fails with `UnsupportedOperation: KeyEncoder for Iceberg format is not yet implemented`.
### Expected behavior
The Rust/Python client can write to Iceberg datalake-enabled tables, matching the Java
client's behavior — i.e. an Iceberg-compatible `KeyEncoder` implementation in
`fluss-rust/crates/fluss/src/row/encode/`.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in fluss-rust/crates/fluss/src/row/encode/mod.rs, where the key-encoder factory currently rejects Iceberg after handling Paimon and Lance. Compare the Rust encoding flow with the Java client's Iceberg-compatible bucket-id behavior, then reproduce the Python write against an Iceberg datalake-enabled primary-key table. Done means the write succeeds without the UnsupportedOperation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100