[Improvement] Spark connector support Paimon distribution (CLUSTERED BY ... INTO ... BUCKETS)
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
Spark Paimon currently does not expose distribution support in connector capability and docs, but the core path already has most of the building blocks.
Background:
- Spark Paimon IT disables clustered-by tests:
- /Users/fanng/opensource/gravitino/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/paimon/SparkPaimonCatalogIT.java:47
- Spark connector already converts bucket transforms to Gravitino distribution and passes it into createTable:
- /Users/fanng/opensource/gravitino/spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTransformConverter.java:140
- /Users/fanng/opensource/gravitino/spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/catalog/BaseCatalog.java:209
- Paimon backend already validates and applies HASH distribution:
- /Users/fanng/opensource/gravitino/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogOperations.java:505
- Current doc says Spark Paimon does not support distribution:
- /Users/fanng/opensource/gravitino/docs/spark-connector/spark-catalog-paimon.md:25
Suggested SQL for validation:
```sql
USE paimon_catalog.default;
CREATE TABLE dist_tbl (
id INT,
name STRING,
address STRING
) USING paimon
CLUSTERED BY (id) INTO 4 BUCKETS;
DESC TABLE EXTENDED dist_tbl;
SHOW TBLPROPERTIES dist_tbl;
```
How should we improve?
- Enable distribution capability for Spark Paimon and add integration coverage.
- Ensure `CLUSTERED BY (...) INTO N BUCKETS` is persisted as Paimon bucket metadata.
- Add/enable ITs to verify table metadata after create/load.
- Update Spark Paimon docs to mark distribution as supported.
Out of scope:
- Sort orders (`SORTED BY`) in this issue.
- Row-level operations (`DELETE/UPDATE/MERGE`) in this issue.
Contributor guide
Research direction
Start with SparkPaimonCatalogIT.java:47 and the current Spark Paimon documentation at spark-catalog-paimon.md:25. Trace bucket conversion in SparkTransformConverter.java:140 and createTable handling in BaseCatalog.java:209, then compare with HASH validation in PaimonCatalogOperations.java:505. Done means the capability is enabled, clustered-by integration tests pass, bucket metadata is verified after create/load, and the documentation is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100