apache / apache/gravitino

[Bug report] ClickHouse SET index parameter is lost during metadata readback

Open
#12,787 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

## Describe what's wrong

The ClickHouse catalog loses the `set_max_values` parameter when loading metadata for a `DATA_SKIPPING_SET` index.

An index created as `TYPE set(100)` is loaded with the correct index type, but without `set_max_values=100` in `Index.properties()`. Recreating a table from the loaded metadata therefore changes the index to `TYPE set(0)`, which may change ClickHouse data-skipping behavior.

## Error message and/or stacktrace

No exception is raised. The parameter is silently lost during metadata readback.

## How to reproduce

1. Use Gravitino `main` and ClickHouse `24.8.14`.
2. Create a table with a SET index:

```sql
CREATE TABLE test_table
(
id UInt64,
value String,
INDEX idx_set value TYPE set(100) GRANULARITY 3
)
ENGINE = MergeTree
ORDER BY id;
```

3. Load the table through the ClickHouse catalog with `Table loaded = tableCatalog.loadTable(NameIdentifier.of("default", "test_table"));`.
4. Inspect `loaded.index()`: the actual SET index properties are `{granularity=3}`, while the expected properties are `{set_max_values=100, granularity=3}`.
5. Recreate another table by passing `loaded.index()` directly to the second `tableCatalog.createTable(...)` call:

```java
Table recreated =
tableCatalog.createTable(
NameIdentifier.of("default", "test_table_recreated"),
loaded.columns(),
loaded.comment(),
loaded.properties(),
loaded.partitioning(),
loaded.distribution(),
loaded.sortOrder(),
loaded.index());
```

6. Compare both tables with `SHOW CREATE TABLE`: the actual recreated table contains `TYPE set(0) GRANULARITY 3`, while the expected definition is `TYPE set(100) GRANULARITY 3`.

## Additional context

Expected behavior:

- `set(100)` loads with `set_max_values=100`;
- `granularity=3` is preserved;
- `set(0)` omits the default `set_max_values` property;
- recreating from loaded metadata preserves `TYPE set(100)`;
- values outside `0..Integer.MAX_VALUE` fail clearly.

The full parameterized index expression is available from the [ClickHouse system.data_skipping_indices.type_full documentation](https://github.com/ClickHouse/ClickHouse/blob/cc8250eae1a9c5569cb5ac226a4633cd9cd2ac3f/docs/reference/system-tables/data_skipping_indices.mdx).

Contributor guide

Open the contributing guide

Research direction

Start at the ClickHouse catalog path used by tableCatalog.loadTable and inspect how DATA_SKIPPING_SET index metadata becomes Index.properties(). Reproduce the CREATE TABLE example, compare loaded.index() with the expected set_max_values and granularity values, then verify completion by recreating the table and comparing SHOW CREATE TABLE output.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, java
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.