[Improvement] ClickHouse catalog: preserve native partition expressions when loading tables
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
The ClickHouse catalog fails to load a valid MergeTree table when its `PARTITION BY`
clause uses an expression outside the current identity/date Transform subset, for
example:
```sql
PARTITION BY cityHash64(toString(sm4_cipher_msg)) % 7
```
The failure blocks ordinary table metadata access. Returning only an empty Transform
array would also be ambiguous because callers could not distinguish an unpartitioned
table from a native expression that Gravitino cannot structure.
#### Current behavior
The load path parses the SHOW CREATE statement before reading the database-qualified
`system.tables.partition_key` fallback. The strict parser raises
`UnsupportedOperationException`, although ClickHouse has already accepted and stored
the expression.
#### How to reproduce
1. Create a table:
```sql
CREATE TABLE hash_partition
(
sm4_cipher_msg String,
payload UInt32
)
ENGINE = MergeTree
ORDER BY tuple()
PARTITION BY cityHash64(toString(sm4_cipher_msg)) % 7;
```
2. Register a Gravitino JDBC ClickHouse catalog pointing to the database.
3. Load the table through the Java API:
```java
catalog.loadTable(NameIdentifier.of("default", "hash_partition"));
```
Alternatively, use the REST API:
```text
GET /api/metalakes/{metalake}/catalogs/{catalog}/schemas/default/tables/hash_partition
```
4. Observe an error response indicating that the partition expression is unsupported.
#### Expected behavior
- `loadTable()` succeeds.
- Every successfully loaded ClickHouse table exposes a visible `partition-key` property.
- For a partitioned table, `partition-key` contains ClickHouse's canonical native
expression as returned by `system.tables.partition_key`.
- For an unpartitioned table, `partition-key` is present with an empty string value.
- `table.properties().get("partition-key")` therefore returns
`cityHash64(toString(sm4_cipher_msg)) % 7` for the example above.
- `table.partitioning()` remains empty for a native expression that has no equivalent
current Transform.
- Identity, `toDate`, `toYear`, and `toYYYYMM` expressions continue returning their
existing structured Transforms while also exposing the canonical native expression.
- Original CREATE statement whitespace and quoting are not guaranteed to be preserved.
### How should we improve?
#### Implementation contract
- Treat `system.tables.partition_key` as the single authoritative source of the
ClickHouse-native partition expression.
- Expose `partition-key` for every loaded ClickHouse table. Use an empty string for an
unpartitioned table so callers can rely on `Map.get(...)` without first calling
`containsKey(...)`.
- Register `partition-key` as a visible, reserved, immutable table property.
- Keep the property output-only: reject create, set, and remove requests for it, and
always filter it from `transformToJdbcProperties()` so it is never written back to
ClickHouse.
- Preserve the existing identity, `toDate`, `toYear`, and `toYYYYMM` Transform mappings,
including tuples composed entirely of supported expressions.
- Make structural conversion best-effort so unsupported valid expressions do not block
`loadTable()`.
- Structure a tuple partition key on an all-or-nothing basis. If any tuple element
cannot be represented, return an empty `partitioning()` array and preserve the full
canonical expression in `partition-key`.
- Require plain-column arguments for the supported date mappings so nested expressions
such as `toYear(toString(event_time))` are not misrepresented as single-field
Transforms.
- Remove the partition-specific SHOW CREATE regex path.
- Preserve `partition-key` and other table properties during JDBC-to-Gravitino property
conversion while retaining the existing `ENGINE` key mapping.
#### Scope and boundaries
This change does not expand the partition expressions or engines supported by the
Gravitino CREATE TABLE API. It only makes loading existing ClickHouse tables
best-effort and preserves their native partition metadata.
This change does not add physical partition listing, partition mutations, a complete
ClickHouse parser, or a new public partitioning type. `partition-key` is
ClickHouse-specific fidelity metadata; it is not the generic Gravitino partitioning API
and does not represent physical partitions.
The change does not add DDL, partition, or engine support for ReplicatedMergeTree
variants, Distributed tables, or other engines that the existing catalog does not
already support.
#### Compatibility and user-facing change
Yes. Every loaded ClickHouse table now exposes a visible, reserved `partition-key`
property:
- A non-empty value is ClickHouse's canonical native partition expression.
- An empty string means that the table is unpartitioned.
Existing structured partitioning JSON and supported DDL behavior remain unchanged.
For native expressions without an equivalent Transform, `Table.partitioning()` is
empty and the full canonical expression remains available through `partition-key`.
The property is read-only: callers cannot supply it during create, set it during alter,
or remove it.
#### Known limitation
A partition key is structured on an all-or-nothing basis. If any element of a tuple
partition key cannot be represented as a supported Transform, `Table.partitioning()`
is empty. Callers can retrieve the full expression from `partition-key`, but cannot use
the structured API to determine which tuple elements were individually representable.
#### References
- [ClickHouse system.tables](https://clickhouse.com/docs/reference/system-tables/tables)
- [ClickHouse MergeTree table engine](https://clickhouse.com/docs/reference/engines/table-engines/mergetree-family/mergetree)
- [Gravitino ClickHouse partition parser](https://github.com/apache/gravitino/blob/main/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java)
- [Gravitino reserved properties](https://github.com/apache/gravitino/blob/main/core/src/main/java/org/apache/gravitino/connector/PropertyEntry.java)
Contributor guide
Research direction
Start with catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java and the ClickHouse catalog loadTable path. Read system.tables.partition_key handling, reserved properties such as PropertyEntry, and transformToJdbcProperties(). Done means loadTable() succeeds for unsupported native expressions, exposes the canonical partition-key property, preserves supported Transforms, and keeps the property read-only and out of JDBC writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100