[spark] Support custom Paimon lake table paths
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
### Motivation
Fluss tables can use `table.datalake.database-name` and
`table.datalake.table-name` to map a Fluss table to a Paimon table with a
different database or table name. https://github.com/apache/fluss/pull/3476
Spark lake reads currently reject tables whose resolved Paimon path differs
from the Fluss table path. This issue tracks adding full Spark support for
custom Paimon lake table paths.
### Scope
- Resolve and use the mapped Paimon database and table name when creating the
Spark lake source.
- Support custom Paimon paths in Fluss/Paimon union reads.
- Ensure predicate pushdown continues to work when the lake source uses a
mapped database or table name.
- Add coverage for custom database mapping, custom table mapping, and combined
database/table mapping.
- Cover both direct lake reads and union reads.
### Current behavior
Spark throws the following exception when a lake read resolves to a different
Paimon table path:
```text
Custom lake table path is not supported for Spark lake reads yet.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Spark lake source creation and the existing rejection for resolved Paimon paths that differ from the Fluss table path. Check direct lake reads and Fluss/Paimon union reads, then add coverage for custom database mapping, custom table mapping, combined mappings, and predicate pushdown; done means all listed reads work with mapped paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100