ClickHouse / ClickHouse/ClickHouse
Parquet filter push-down compares raw file values while a narrowing schema hint changes the decoded value: `WHERE` silently loses rows that `SELECT` shows, at default settings
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Describe what's wrong
When a Parquet file is read with a schema hint whose decode is not injective on the file's physical values — a narrowing integer hint (`UInt16` over an `INT32` column) or a `Date`/`DateTime` hint over a plain integer column holding out-of-range values — the data path collapses out-of-range raw values into in-range ones (truncation for narrowing integer hints; saturation for `DateTime`), but every filter push-down leg keeps comparing the query constant against the raw file values. The result is that `SELECT` shows a row while `WHERE` on exactly the value shown silently returns nothing, at default settings.
All four legs are independently affected on the narrowing-hint arm: row-group min/max statistics (`input_format_parquet_filter_push_down`), the page index (`input_format_parquet_page_filter_push_down`), the bloom filter (`input_format_parquet_bloom_filter_push_down`), and the dictionary filter (`input_format_parquet_dictionary_filter_push_down`). Turning off any three still loses the row; only turning off all four restores it. The pre-v3 arrow reader (`input_format_parquet_use_native_reader_v3 = 0`) loses the row as well.
The files below are written by ClickHouse itself, so it cannot round-trip its own output under such a hint with a point predicate.
### Does it reproduce on the most recent release?
Yes — reproduced identically on 26.8.2.7 and on a current master build.
### How to reproduce
Narrowing integer hint (truncation): `65541 = 65536 + 5` decodes as `5` under a `UInt16` hint, but no filter leg knows that.
```
clickhouse-local -q "SELECT arrayJoin([toInt32(65541), toInt32(7)]) AS x INTO OUTFILE 'u16.parquet' FORMAT Parquet"
clickhouse-local -q "SELECT x FROM file('u16.parquet', Parquet, 'x UInt16') ORDER BY x"
-- 5
-- 7
clickhouse-local -q "SELECT count() FROM file('u16.parquet', Parquet, 'x UInt16') WHERE x = 5"
-- 0
clickhouse-local -q "SELECT count() FROM file('u16.parquet', Parquet, 'x UInt16') WHERE x = 5 SETTINGS input_format_parquet_filter_push_down = 0, input_format_parquet_page_filter_push_down = 0, input_format_parquet_bloom_filter_push_down = 0, input_format_parquet_dictionary_filter_push_down = 0"
-- 1
```
`DateTime` hint over epoch seconds with a `-1` sentinel (saturates to `1970-01-01 00:00:00` on decode):
```
clickhouse-local -q "SELECT arrayJoin([toInt64(-1), toInt64(1700000000)]) AS ts INTO OUTFILE 'ts64.parquet' FORMAT Parquet"
clickhouse-local -q "SELECT ts FROM file('ts64.parquet', Parquet, 'ts DateTime') ORDER BY ts"
-- 1970-01-01 00:00:00
-- 2023-11-14 22:13:20
clickhouse-local -q "SELECT count() FROM file('ts64.parquet', Parquet, 'ts DateTime') WHERE ts = toDateTime(0)"
-- 0 (1 with the four push-down settings above turned off)
```
Same shape with a `Date` hint over `INT32` holding `-5`: the row decodes and displays as `1970-01-01`, `WHERE x = '1970-01-01'` returns nothing at defaults. On that arm the row-group statistics leg alone happens to be fail-safe (the negative endpoint is dropped), but the page-index, bloom-filter and dictionary-filter legs each lose the row on their own.
Controls: with in-range values the same hints filter correctly (`WHERE x = 7` returns 1), and reading without a hint filters correctly (`WHERE x = 65541` returns 1), so this is specifically the interaction of the hint conversion with the push-down legs.
### Expected behavior
A query with `WHERE x = c` returns exactly the rows in which the decoded column — what `SELECT` shows — equals `c`.
### Additional context
Mechanism, current master:
* The row-group and page-index legs build `Range` endpoints from the raw statistics bytes via `IntConverter::convertField` (`src/Processors/Formats/Impl/Parquet/Decoding.cpp:1660`); for a `UInt16` hint over `INT32` the endpoints stay in raw space (e.g. `[7, 65541]`), so a query constant that is only reachable through truncation (`5`) falls outside them and the row group / page is pruned.
* The bloom-filter and dictionary-filter legs hash the query constant with `parquetTryHashField` against the physical column descriptor (`src/Processors/Formats/Impl/Parquet/Reader.cpp:935`); the file's bloom filter and dictionary contain hashes of the raw values (`65541`, `7`), so the constant `5` misses.
* The data path, by contrast, decodes through the hint: a narrowing integer hint copies the low bytes (`IntConverter::convertColumn`, `src/Processors/Formats/Impl/Parquet/Decoding.cpp:1618-1623`), and `Date`/`DateTime` hints saturate out-of-range values. The stats dispatch (`dispatch_int_stats_converter`, `src/Processors/Formats/Impl/Parquet/SchemaConverter.cpp:924`) allows statistics for all of these output types without accounting for the value collapse.
This is a different defect from #118376 (there the hint flips the signedness so the declared bound *order* is wrong; here the hint's value *mapping* is non-injective, and the bloom/dictionary legs — which #118376 does not affect — lose rows too). The dictionary leg already contains precedent for exactly this class of reasoning: under `input_format_null_as_default` it adds the output type's default-value hash because nulls decode to a value that is not in the dictionary. The hint conversions create the same situation for non-null values.
Contributor guide
Assessment
This issue has not been assessed yet.