ClickHouse / ClickHouse/ClickHouse

Iceberg min/max pruning reads UUID manifest bounds without the RFC byte swap: WHERE over a UUID column silently misses rows

Open
#118,366 2 comments 0 reactions 1 assignee Claimed by @scanhex12 View on GitHub
bug comp-datalake
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

ClickHouse

### Describe what's wrong

When Iceberg data-file pruning evaluates a predicate over a `UUID` column, the manifest's `lower_bounds`/`upper_bounds` payloads are copied into the range verbatim, without the RFC-4122 byte-order conversion that the Parquet data path applies. The Iceberg spec serializes `uuid` bounds as 16 big-endian bytes, and the data path decodes them by reversing each 8-byte half (`readColumnWithUUIDFromFixedBinaryData`, `src/Processors/Formats/Impl/ArrowColumnToCHColumn.cpp:1212`, and the native Parquet reader equivalently), which is also how a `UUID` literal parses. But `deserializeFieldFromBinaryRepr` in `src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp:142` takes the "binary representation matches our internal representation" branch and does a raw `insertData` of the 16 big-endian bytes.

So the min/max range that `ManifestFilesPruner` checks lives in a different value space than the query constant, the containment test fails for most values, and the data file holding the matching rows is silently pruned: `SELECT ... WHERE u = ''` returns fewer rows than the table contains, with exit code 0. It happens at pure default settings on tables written by standard external writers (pyiceberg, Spark), which write `uuid` bounds by default. Whether a given value survives is data-dependent (a value whose 8-byte halves are palindromic round-trips unchanged and is pruned correctly), so the row loss is intermittent by value, and the mangled pair can also arrive inverted, in which case the inverted-bounds guard happens to disable pruning.

### Does it reproduce on the most recent release?

Yes. Reproduced on 26.8.2.7 (official LTS binary) and on a current 26.9.1.1 development build.

### How to reproduce

Create an Iceberg table with a `UUID` column using pyiceberg (0.11.1, pyarrow 19), one data file per row so each file's manifest bound pair is `lower == upper == `:

```python
import shutil, uuid as uuidlib
from pathlib import Path
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, IntegerType, UUIDType
from pyiceberg.io.pyarrow import schema_to_pyarrow

BASE = Path("/path/inside/user_files/wh") # must be inside the server's user_files_path
BASE.mkdir(parents=True)
catalog = SqlCatalog("local", uri=f"sqlite:///{BASE}/catalog.db", warehouse=f"file://{BASE}")
catalog.create_namespace("db")
schema = Schema(
NestedField(1, "id", IntegerType(), required=False),
NestedField(2, "u", UUIDType(), required=False),
)
tbl = catalog.create_table("db.t", schema=schema)
arrow_schema = schema_to_pyarrow(schema)

U_A = uuidlib.UUID("00112233-4455-6677-8899-aabbccddeeff") # ordinary value
U_C = uuidlib.UUID("00112233-3322-1100-8899-aabbbbaa9988") # control: each 8-byte half is a palindrome

for i, u in ((1, U_A), (2, U_C)):
storage = pa.array([u.bytes], pa.binary(16))
u_arr = pa.ExtensionArray.from_storage(arrow_schema.field("u").type, storage)
tbl.append(pa.Table.from_arrays([pa.array([i], arrow_schema.field("id").type), u_arr], schema=arrow_schema))
```

The manifests then carry the spec-mandated big-endian bytes (dumped with pyiceberg): file 1 bounds `00112233445566778899aabbccddeeff`, file 2 bounds `00112233332211008899aabbbbaa9988`.

Queries against a stock server (all settings at defaults):

```sql
SELECT count() FROM icebergLocal('/wh/db/t');
-- 2

SELECT countIf(u = toUUID('00112233-4455-6677-8899-aabbccddeeff')) FROM icebergLocal('/wh/db/t');
-- 1 (ground truth: no WHERE, so no pruning; the data path decodes the value correctly)

SELECT count() FROM icebergLocal('/wh/db/t') WHERE u = toUUID('00112233-4455-6677-8899-aabbccddeeff');
-- 0 <-- wrong: the file containing the row was pruned

SELECT count() FROM icebergLocal('/wh/db/t') WHERE u = toUUID('00112233-3322-1100-8899-aabbbbaa9988');
-- 1 (control: byte-swap-invariant value, pruning bound accidentally equals the canonical value)

SELECT count() FROM icebergLocal('/wh/db/t') WHERE id = 1;
-- 1 (control: integer bounds prune correctly, the pruning machinery itself is fine)
```

`system.events` confirms the attribution: the wrong query increments `IcebergMinMaxIndexPrunedFiles` by 2 — both data files pruned, zero files read.

### Expected behavior

`SELECT count() ... WHERE u = toUUID('00112233-4455-6677-8899-aabbccddeeff')` returns 1: the manifest bound bytes are decoded with the same RFC-4122 byte-order conversion as the data path, so min/max pruning over a `UUID` column keeps every file whose range contains the queried value.

### Error message and/or stacktrace

None: the query succeeds with exit code 0 and returns too few rows.

### Additional context

`deserializeFieldFromBinaryRepr` funnels every non-Decimal fixed-width type through the raw-copy branch. For the little-endian integer/float/date types that is correct, and #118364 adds a width guard to the same function, but `uuid` is the one big-endian fixed-width type in the Iceberg spec (Appendix D: "16-byte big-endian value"), so the raw copy lands in `ColumnVector` with the halves unswapped. A fix would apply the same per-half reversal that `readColumnWithUUIDFromFixedBinaryData` applies before constructing the range endpoints (both call sites: the column bounds used by `ManifestFilesPruner`, and the row-lineage bounds).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.