apache / apache/gravitino

[Improvement] Support ClickHouse data-skipping indexes and engine-aware sort orders validation in CreateTableDialog

Open
#12,337 0 comments 0 reactions 1 assignee Claimed by @LauraXia123 View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

## Description

The `CreateTableDialog` in the Gravitino web UI needs two enhancements for ClickHouse table creation:

### 1. Support ClickHouse data-skipping index types with properties

ClickHouse supports data-skipping indexes (`data_skipping_minmax`, `data_skipping_bloom_filter`, `data_skipping_set`) which have additional properties:

- **Granularity** - applies to all data-skipping index types
- **Set Max Values** - applies only to `data_skipping_set`

Currently, the Indexes tab does not render these property fields. The table creation form should:

- Show a **Granularity** column for data-skipping indexes (hidden for `primary_key`)
- Show a **Set Max Values** column only for `data_skipping_set` index type
- Populate these properties when editing an existing table
- Include index properties in the submit payload

### 2. Engine-aware sort orders validation

Sort orders are required for ClickHouse MergeTree family engines (`MergeTree`, `ReplacingMergeTree`, `SummingMergeTree`, etc.), but **not** required for other ClickHouse engine types (`Distributed`, `TinyLog`, `Log`, `StripeLog`).

Currently, the sort orders validation does not adapt when the user switches the engine type. This causes:

- Sort Orders tab always shows the required indicator (`*`) regardless of engine type
- Form validation fails for non-MergeTree engines when sort orders are empty
- Switching from a MergeTree engine to a non-MergeTree engine does not clear residual sort orders data or validation errors

The expected behavior:

- **MergeTree family engines**: Sort Orders tab shows `*` indicator, sort orders are required, nullable columns cannot be used in ORDER BY
- **Non-MergeTree engines** (Distributed, TinyLog, Log, StripeLog): Sort Orders tab does not show `*` indicator, sort orders are optional
- **Engine switch**: When switching from MergeTree to non-MergeTree, clear sort orders data and validation errors; when switching to MergeTree, show required validation if sort orders are empty

## Affected files

- `web-v2/web/src/app/catalogs/rightContent/CreateTableDialog.js`
- `web-v2/web/src/config/index.js` (already has `clickHouseMergeTreeEngines` and `indexesInfoMap` for `jdbc-clickhouse`)

## Implementation notes

- Sort orders validation is moved from `Form.Item rules` (which can cache stale closure values) to `handleSubmit` where `form.getFieldValue('engine')` is called to read the current engine value in real-time
- A `useEffect` on `isClickHouseMergeTreeEngine` handles real-time UI updates when the engine type changes
- Data-skipping index properties (granularity, set_max_values) are conditionally rendered using `Form.Item shouldUpdate` and persisted in the index `properties` object

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.