ClickHouse / ClickHouse/ClickHouse

Iceberg table with a geometry column becomes unreadable after server restart; allow_experimental_geo_types_in_iceberg is latched at ATTACH, not checked per query

Open
#115,082 3 comments 0 reactions 1 assignee Claimed by @scanhex12 View on GitHub
bug-unreleased comp-datalake
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Describe what's wrong**

The experimental gate `allow_experimental_geo_types_in_iceberg` is evaluated against the context captured when the Iceberg storage object is constructed (CREATE / ATTACH / server startup), not against the query context. Two user-visible consequences:

1. **An Iceberg table with a `geometry` column becomes permanently unreadable after a server restart** (or after any `DETACH`/`ATTACH` without the flag in the session): every subsequent `SELECT` throws `Code: 36` — *even when the query itself sets `allow_experimental_geo_types_in_iceberg = 1`*. The only cure is a manual `DETACH TABLE` + `ATTACH TABLE` with the flag enabled in the session.
2. Conversely, a table attached with the flag enabled is freely readable by queries that do **not** set the flag, so the gate is not actually enforced at read time.

The gate lives in `IcebergSchemaProcessor` (`src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp`, the `allow_geo_parser` member checked when parsing a `geometry`/`geography` field), and `allow_geo_parser` is fixed from the settings of whichever context built the metadata object. Server startup attaches tables with server-default settings, so restart wedges the table. This is the same constructor-latched-guard shape as #115044, but with a worse consequence: a restart breaks working tables.

**How to reproduce**

ClickHouse 26.8.1.1516 (current master).

```sql
SET allow_experimental_geo_types_in_iceberg = 1;

CREATE TABLE geo (id Int64, g Geometry) ENGINE = IcebergLocal('/var/lib/clickhouse/user_files/geo/', 'Parquet');

INSERT INTO geo SELECT 1, readWKT('POINT(1 2)') SETTINGS allow_insert_into_iceberg = 1;

SELECT id, wkt(g) FROM geo; -- works: 1 POINT(1 2)
```

Now restart the server (or, in the same session: `SET allow_experimental_geo_types_in_iceberg = 0; DETACH TABLE geo; ATTACH TABLE geo;`), then:

```sql
SELECT id, wkt(g) FROM geo SETTINGS allow_experimental_geo_types_in_iceberg = 1;
```

```
Code: 36. DB::Exception: Using geometry/geography types is not allowed without enabled allow_experimental_geo_types_in_iceberg flag. (BAD_ARGUMENTS)
```

The flag on the query does not help. Recovery:

```sql
SET allow_experimental_geo_types_in_iceberg = 1;
DETACH TABLE geo;
ATTACH TABLE geo;
SELECT id, wkt(g) FROM geo; -- works again, and keeps working even for queries WITHOUT the flag
```

**Expected behavior**

Either the gate is enforced per query (a read without the flag fails, a read with the flag succeeds, regardless of how the storage was constructed), or — if the gate is intentionally a table-level property — it should be captured at CREATE and survive ATTACH/restart. A working table must not become unreadable after a restart while the documented enabling flag is powerless to read it.

**Additional context**

`Related: https://github.com/ClickHouse/ClickHouse/issues/115044` (the same constructor-only-guard shape on the Delta Lake CDF settings).

Found by an automatic optimizer-testing framework (differential testing of optimizer settings, query plans, and equivalent rewrites).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.