apache / apache/fluss

[rust/python] Writing to datalake-enabled Iceberg tables fails: "KeyEncoder for Iceberg format is not yet implemented"

Open
#3,818 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the issues and found nothing similar.

### Fluss version

0.9.1-incubating (server), pyfluss 0.1.0 (client). The gap is still present on current `main`.

### Please describe the bug 🐞

Any write to a primary-key table with `table.datalake.enabled = 'true'` and Iceberg as the
cluster's datalake format fails from the Python client (`pyfluss`, backed by `fluss-rust`)
with:

```
KeyEncoder for Iceberg format is not yet implemented
```

The same write against the same table succeeds with the Java client / Flink connector, so
the table itself is fine — the limitation is client-side.

Root cause: in `fluss-rust/crates/fluss/src/row/encode/mod.rs`, the key-encoder factory
implements Paimon and Lance but explicitly rejects Iceberg:

```rust
match data_lake_format {
Some(DataLakeFormat::Paimon) => {
Ok(Box::new(PaimonKeyEncoder::new(row_type, key_fields)?))
}
Some(DataLakeFormat::Lance) => Ok(Box::new(CompactedKeyEncoder::create_key_encoder(
row_type, key_fields,
)?)),
Some(DataLakeFormat::Iceberg) => Err(Error::UnsupportedOperation {
message: "KeyEncoder for Iceberg format is not yet implemented".to_string(),
}),
None => Ok(Box::new(CompactedKeyEncoder::create_key_encoder(
row_type, key_fields,
)?)),
}
```

The Java client encodes keys for Iceberg-enabled tables (bucket-id computation compatible
with Iceberg's bucket transform), so the Rust client needs the equivalent encoder to reach
parity.

### Impact

This blocks Python-based pipelines (e.g. CDC ingestion via pyfluss) from writing to any
datalake-enabled table on Iceberg clusters — the datalake feature and the Python client are
mutually exclusive today. Reported by a production user running Fluss 0.9.1-incubating with
pyfluss 0.1.0.

### Steps to reproduce

1. Run a Fluss cluster with `datalake.format: iceberg`.
2. Create a primary-key table with `'table.datalake.enabled' = 'true'`.
3. From Python: connect with pyfluss 0.1.0 and attempt any upsert into the table.
4. The write fails with `UnsupportedOperation: KeyEncoder for Iceberg format is not yet implemented`.

### Expected behavior

The Rust/Python client can write to Iceberg datalake-enabled tables, matching the Java
client's behavior — i.e. an Iceberg-compatible `KeyEncoder` implementation in
`fluss-rust/crates/fluss/src/row/encode/`.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in fluss-rust/crates/fluss/src/row/encode/mod.rs, where the key-encoder factory currently rejects Iceberg after handling Paimon and Lance. Compare the Rust encoding flow with the Java client's Iceberg-compatible bucket-id behavior, then reproduce the Python write against an Iceberg datalake-enabled primary-key table. Done means the write succeeds without the UnsupportedOperation error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.