[Feature] Hope to retrieve FileStoreTable information via HMS in SparkSource#loadTable.
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
In the current implementation, when writing to an existing Paimon table via Spark DataFrame (e.g., df.write.format("paimon").mode("append").option("path",xxx).save()), the SparkTable initialization can only create a FileStoreTable instance through CoreOptions.path(options), even if a Hive Catalog has already been configured.
While using table-path for initialization is ok, there exists a scenario where the path of an existing Paimon table does not conform to the standard path format. This results in validation failure and write errors, with the following message:
"Path 'xxx' is not a valid path. Please use a catalog table path instead: 'warehouse_path/your_database.db/your_table'."
### Solution
Add a method in CoreOptions to retrieve TABLE_ID (the full name of the table), then instantiate a FileStoreTable via HiveCatalog to resolve the aforementioned issues.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at SparkSource#loadTable and inspect how SparkTable currently creates FileStoreTable through CoreOptions.path(options). Then trace CoreOptions and HiveCatalog around TABLE_ID lookup; done means an existing table with a nonstandard path can be initialized through HMS without the current path validation failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100