apache / apache/paimon

[Feature] Support DelegatingCatalogExtension in Paimon SparkCatalog for unified catalog integration

Open
#7,629 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Hi Paimon community, I'm currently working on integrating Apache Paimon into the Apache Polaris Spark catalog.
During this integration, we noticed an architectural difference in how Paimon handles **Spark catalog operations** compared to other formats like Delta and Hudi.
In unified catalogs like Polaris, Delta and Hudi achieve seamless integration by extending Spark's `DelegatingCatalogExtension`.
This allows them to delegate general catalog operations (such as namespace management and table listing) to the underlying unified catalog, while continuing to handle format-specific operations themselves.

## **The Current Challenge:**
Because Paimon's SparkCatalog does not implement this delegation model and directly manages its own warehouse, we cannot configure Polaris as a delegate catalog for Paimon. Instead, we have to rely on a complex hybrid workaround:

- We route Paimon operations through a separate catalog instance to handle operations directly against a segregated Paimon warehouse.

- After a table is created, we must manually register it as an external table back into Polaris so it appears in the unified catalog view.

## **The Request:**
Would the Paimon community be open to supporting `DelegatingCatalogExtension` (or a similar delegation architecture) in its SparkCatalog?

I'd love to hear your thoughts on the feasibility of this and if it's something that could be considered for the roadmap.

Thanks!

## **Reference:**
[Add Apache Paimon support to Polaris Spark Catalog](https://github.com/apache/polaris/pull/3723)
However, while working on the integration tests, I found out that the delegating catalog does not work.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading SparkCatalog and Spark's DelegatingCatalogExtension, then review the referenced Polaris integration to understand the delegation expectations. Define how general catalog operations should delegate while Paimon-specific operations remain local, and validate the behavior with integration tests; done means Paimon works as a delegate in unified catalog integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.