apache / apache/gravitino

[Improvement] Spark connector support Paimon distribution (CLUSTERED BY ... INTO ... BUCKETS)

Open
#10,367 2 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

Spark Paimon currently does not expose distribution support in connector capability and docs, but the core path already has most of the building blocks.

Background:
- Spark Paimon IT disables clustered-by tests:
- /Users/fanng/opensource/gravitino/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/paimon/SparkPaimonCatalogIT.java:47
- Spark connector already converts bucket transforms to Gravitino distribution and passes it into createTable:
- /Users/fanng/opensource/gravitino/spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTransformConverter.java:140
- /Users/fanng/opensource/gravitino/spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/catalog/BaseCatalog.java:209
- Paimon backend already validates and applies HASH distribution:
- /Users/fanng/opensource/gravitino/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogOperations.java:505
- Current doc says Spark Paimon does not support distribution:
- /Users/fanng/opensource/gravitino/docs/spark-connector/spark-catalog-paimon.md:25

Suggested SQL for validation:

```sql
USE paimon_catalog.default;

CREATE TABLE dist_tbl (
id INT,
name STRING,
address STRING
) USING paimon
CLUSTERED BY (id) INTO 4 BUCKETS;

DESC TABLE EXTENDED dist_tbl;
SHOW TBLPROPERTIES dist_tbl;
```

How should we improve?
- Enable distribution capability for Spark Paimon and add integration coverage.
- Ensure `CLUSTERED BY (...) INTO N BUCKETS` is persisted as Paimon bucket metadata.
- Add/enable ITs to verify table metadata after create/load.
- Update Spark Paimon docs to mark distribution as supported.

Out of scope:
- Sort orders (`SORTED BY`) in this issue.
- Row-level operations (`DELETE/UPDATE/MERGE`) in this issue.

Contributor guide

Open the contributing guide

Research direction

Start with SparkPaimonCatalogIT.java:47 and the current Spark Paimon documentation at spark-catalog-paimon.md:25. Trace bucket conversion in SparkTransformConverter.java:140 and createTable handling in BaseCatalog.java:209, then compare with HASH validation in PaimonCatalogOperations.java:505. Done means the capability is enabled, clustered-by integration tests pass, bucket metadata is verified after create/load, and the documentation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.