apache / apache/hudi

Archival considers retaining the `the earliest retain instant` in the clean plan

Open
#17,109 2 comments 0 reactions 1 assignee Claimed by @danny0405 View on GitHub
from-jira priority:high status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

[https://github.com/apache/hudi/pull/13558#discussion_r2216883612]

There is a specific check on Polaris catalog class.   In the future, can this be change to a general check on V2 catalog implementation, instead of a specific catalog implementation class like Polaris, so other V2 catalog implementation class can also be used?

When I mentioned "general check on V2 catalog implementation", I was referring to a check on {{DelegatingCatalogExtension}} implementation that returns v2 table, i.e., {{org.apache.spark.sql.connector.catalog.Table}} introduced in Spark 3, and such a catalog implementation is specified through the Spark config {{{}spark.sql.catalog.=...{}}}, which is how {{org.apache.polaris.spark.SparkCatalog}} is plugged in.

The complication in Hudi Spark Implementation is that the Spark Datasource v1 is used for Hudi read and write, and the {{HoodieCatalog}} has a mixed mode of using v1 table ({{{}org.apache.spark.sql.catalyst.catalog.CatalogTable{}}}) and v2 table. We should revisit the check to see if it can be made general.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9634
- Type: Improvement
- Fix version(s):
- 1.2.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.