ClickHouse / ClickHouse/ClickHouse
nested array inference
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 17h 34m
- Merged PRs (30d)
- 479
Description
### Company or project name
_No response_
### Describe the unexpected behaviour
I am trying to ingest data from a parquet file where I have two array columns named like a.x and a.y. I believe clickhouse is interpreting these as nested columns and validating they have the same size (which I do not want it to do). I have found no way to disable this behaviour and it is very difficult for me to change these channel names programatically (we ingest them using the s3 table function).
is there some feature I'm perhaps missing or a suggested way to get around this limitation. if not, how hard do you anticipate it being to add such a setting myself
### Which ClickHouse versions are affected?
24.x, 25.x at least
### How to reproduce
ingest a parquet with two array columns (a.x, a.y) where they have different sized arrays
### Expected behavior
a setting to control the nesting inference
### Error message and/or stacktrace
(Array columns) have different array sizes. (SIZES_OF_ARRAYS_DONT_MATCH)
### Additional context
_No response_
Contributor guide
Research direction
Start by reproducing the issue with a Parquet file containing array columns named a.x and a.y whose arrays have different lengths, using the s3 table function. Trace how nested-column inference and array-size validation are selected. Done means a setting controls that inference and the mismatched arrays can be ingested when nesting is disabled, with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100