apache / apache/paimon

[Feature] Hope to retrieve FileStoreTable information via HMS in SparkSource#loadTable.

Open
#5,770 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

In the current implementation, when writing to an existing Paimon table via Spark DataFrame (e.g., df.write.format("paimon").mode("append").option("path",xxx).save()), the SparkTable initialization can only create a FileStoreTable instance through CoreOptions.path(options), even if a Hive Catalog has already been configured.

While using table-path for initialization is ok, there exists a scenario where the path of an existing Paimon table does not conform to the standard path format. This results in validation failure and write errors, with the following message:

"Path 'xxx' is not a valid path. Please use a catalog table path instead: 'warehouse_path/your_database.db/your_table'."

### Solution

Add a method in CoreOptions to retrieve TABLE_ID (the full name of the table), then instantiate a FileStoreTable via HiveCatalog to resolve the aforementioned issues.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at SparkSource#loadTable and inspect how SparkTable currently creates FileStoreTable through CoreOptions.path(options). Then trace CoreOptions and HiveCatalog around TABLE_ID lookup; done means an existing table with a nonstandard path can be initialized through HMS without the current path validation failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.