[Bug report] ClickHouse SET index parameter is lost during metadata readback
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
## Describe what's wrong
The ClickHouse catalog loses the `set_max_values` parameter when loading metadata for a `DATA_SKIPPING_SET` index.
An index created as `TYPE set(100)` is loaded with the correct index type, but without `set_max_values=100` in `Index.properties()`. Recreating a table from the loaded metadata therefore changes the index to `TYPE set(0)`, which may change ClickHouse data-skipping behavior.
## Error message and/or stacktrace
No exception is raised. The parameter is silently lost during metadata readback.
## How to reproduce
1. Use Gravitino `main` and ClickHouse `24.8.14`.
2. Create a table with a SET index:
```sql
CREATE TABLE test_table
(
id UInt64,
value String,
INDEX idx_set value TYPE set(100) GRANULARITY 3
)
ENGINE = MergeTree
ORDER BY id;
```
3. Load the table through the ClickHouse catalog with `Table loaded = tableCatalog.loadTable(NameIdentifier.of("default", "test_table"));`.
4. Inspect `loaded.index()`: the actual SET index properties are `{granularity=3}`, while the expected properties are `{set_max_values=100, granularity=3}`.
5. Recreate another table by passing `loaded.index()` directly to the second `tableCatalog.createTable(...)` call:
```java
Table recreated =
tableCatalog.createTable(
NameIdentifier.of("default", "test_table_recreated"),
loaded.columns(),
loaded.comment(),
loaded.properties(),
loaded.partitioning(),
loaded.distribution(),
loaded.sortOrder(),
loaded.index());
```
6. Compare both tables with `SHOW CREATE TABLE`: the actual recreated table contains `TYPE set(0) GRANULARITY 3`, while the expected definition is `TYPE set(100) GRANULARITY 3`.
## Additional context
Expected behavior:
- `set(100)` loads with `set_max_values=100`;
- `granularity=3` is preserved;
- `set(0)` omits the default `set_max_values` property;
- recreating from loaded metadata preserves `TYPE set(100)`;
- values outside `0..Integer.MAX_VALUE` fail clearly.
The full parameterized index expression is available from the [ClickHouse system.data_skipping_indices.type_full documentation](https://github.com/ClickHouse/ClickHouse/blob/cc8250eae1a9c5569cb5ac226a4633cd9cd2ac3f/docs/reference/system-tables/data_skipping_indices.mdx).
Contributor guide
Research direction
Start at the ClickHouse catalog path used by tableCatalog.loadTable and inspect how DATA_SKIPPING_SET index metadata becomes Index.properties(). Reproduce the CREATE TABLE example, compare loaded.index() with the expected set_max_values and granularity values, then verify completion by recreating the table and comparing SHOW CREATE TABLE output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100