ClickHouse / ClickHouse/ClickHouse

Direct `Map` subcolumn reads (`m.keys` / `m.values`) from Parquet silently return default values

Open
#113,976 2 comments 0 reactions 0 assignees View on GitHub
comp-formats unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

🕵 (Found while working on #104431.)

**Describe what's wrong**

Reading the `keys` / `values` subcolumns of a `Map` column directly from a Parquet file silently returns default values (empty arrays) instead of the data, because the format-level readers treat `m.keys` / `m.values` as columns missing from the file, and `input_format_parquet_allow_missing_columns = 1` (the default) fills them with defaults.

```sql
INSERT INTO FUNCTION file('map_bug.parquet')
SELECT map('v', 'x') AS m FROM numbers(3)
SETTINGS engine_file_truncate_on_insert = 1;

SELECT m.keys, m.values FROM file('map_bug.parquet');
```

returns

```
[] []
[] []
[] []
```

With `input_format_parquet_allow_missing_columns = 0` the mechanism is visible:

```
Code: 8. DB::Exception: Column m.keys was not found in parquet schema: ... (THERE_IS_NO_COLUMN)
```

The same query against a `MergeTree` table returns `['v'] ['x']` as expected. Whole-map access (`m['v']`, `mapKeys(m)`, `mapValues(m)`) over the same Parquet file returns correct data, and addressable `Tuple` subcolumns inside arrays (e.g. `a.x` for `a Array(Tuple(x UInt64, s String))`) are read correctly, so this is specific to the `Map` `keys` / `values` subcolumns.

**Does it reproduce on the most recent release?**

Yes — reproduced on `26.7.1.1362` and current `master`, with both `input_format_parquet_use_native_reader_v3 = 1` (`ParquetV3BlockInputFormat`) and the arrow-based reader (`input_format_parquet_use_native_reader_v3 = 0`), and with both an inferred and an explicitly specified schema.

**Expected behavior**

Either the subcolumn read is served from the file (the map's footer leaves `m.key_value.key` / `m.key_value.value` hold exactly the requested data; the reader already renames the map tuple elements to `keys` / `values` in `SchemaConverter` when reading the whole map), or the storage should not claim subcolumn support it cannot serve, so the query falls back to reading the whole column and extracting the subcolumn — anything but silently returning wrong (default) values for a column that exists in the file.

Contributor guide

Open the contributing guide

Research direction

Reproduce the SQL example against the Parquet file with both values of input_format_parquet_use_native_reader_v3 and with missing columns allowed and disallowed. Then read ParquetV3BlockInputFormat, the arrow-based reader, and SchemaConverter to trace how m.keys and m.values are resolved versus m.key_value.key and m.key_value.value. Done means direct Map subcolumn reads return the stored keys and values, or unsupported subcolumns fall back to whole-map extraction without default-value results.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.