apache / apache/fluss

[spark] Support custom Paimon lake table paths

Open
#3,832 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

### Motivation

Fluss tables can use `table.datalake.database-name` and
`table.datalake.table-name` to map a Fluss table to a Paimon table with a
different database or table name. https://github.com/apache/fluss/pull/3476

Spark lake reads currently reject tables whose resolved Paimon path differs
from the Fluss table path. This issue tracks adding full Spark support for
custom Paimon lake table paths.

### Scope

- Resolve and use the mapped Paimon database and table name when creating the
Spark lake source.
- Support custom Paimon paths in Fluss/Paimon union reads.
- Ensure predicate pushdown continues to work when the lake source uses a
mapped database or table name.
- Add coverage for custom database mapping, custom table mapping, and combined
database/table mapping.
- Cover both direct lake reads and union reads.

### Current behavior

Spark throws the following exception when a lake read resolves to a different
Paimon table path:

```text
Custom lake table path is not supported for Spark lake reads yet.

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Spark lake source creation and the existing rejection for resolved Paimon paths that differ from the Fluss table path. Check direct lake reads and Fluss/Paimon union reads, then add coverage for custom database mapping, custom table mapping, combined mappings, and predicate pushdown; done means all listed reads work with mapped paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.