[lake/iceberg] Support custom Iceberg lake table path
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Fluss currently assumes that an Iceberg lake table uses the same database and table name as the corresponding Fluss table. In practice, an existing Iceberg table may use a different database or table name because of naming conventions, migration requirements, or catalog organization.
Fluss already provides the generic table options `table.datalake.database-name` and `table.datalake.table-name` for mapping a Fluss table to a custom Paimon table path. However, these options are currently rejected for Iceberg. In addition, the Iceberg tiering validator, writer, and committer still load the Iceberg table using the Fluss table path.
We should support mapping a Fluss table to an Iceberg table with a different database and/or table name while preserving the Fluss table path as the logical identity used for table metadata, lake snapshots, and tiering offsets.
### Solution
Reuse the existing generic mapping options instead of introducing Iceberg-specific options:
```sql
'table.datalake.database-name' = 'iceberg_database',
'table.datalake.table-name' = 'iceberg_table'
```
Proposed scope:
- Allow `table.datalake.database-name` and `table.datalake.table-name` for Iceberg tables.
- Fall back to the Fluss database or table name when either option is omitted.
- Resolve the mapped Iceberg path through `TableInfo#getLakeTablePath()` in the Iceberg tiering table validator, writer, and committer.
- Keep the Fluss table path and table ID unchanged for Fluss metadata, lake snapshot tracking, and bucket offsets.
- Treat the mapping options as create-time-only after the Iceberg table has been created or bound, following the existing Paimon behavior. Reject attempts to rebind an existing Fluss table to another Iceberg table.
- Verify Flink `$lake` reads, metadata-table reads, union reads, and historical partition lookups against the mapped Iceberg table.
- Add coverage for custom database mapping, custom table mapping, combined database/table mapping, existing compatible Iceberg tables, tiering, union reads, schema evolution, and invalid mapping alterations.
- Preserve the current same-name behavior when no mapping option is configured.
Spark support for custom lake table paths can remain outside this issue because Spark currently rejects custom lake paths explicitly and requires separate reader changes.
### Anything else?
The Coordinator and Flink lake-source paths already have common lake-table path resolution. The main Iceberg-specific gap is the tiering path and create-time mapping validation. No new Iceberg-specific configuration key should be necessary.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing TableInfo#getLakeTablePath() into the Iceberg tiering table validator, writer, and committer. Review the existing create-time mapping validation and Flink $lake, metadata-table, union, and historical-partition read coverage. Done means custom database/table mappings work with tiering and reads, same-name behavior remains unchanged, and invalid mapping alterations are rejected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100