ClickHouse / ClickHouse/ClickHouse

nested array inference

Open
#86,835 1 comment 0 reactions 0 assignees View on GitHub
comp-formats external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
17h 34m
Merged PRs (30d)
479

Description

### Company or project name

_No response_

### Describe the unexpected behaviour

I am trying to ingest data from a parquet file where I have two array columns named like a.x and a.y. I believe clickhouse is interpreting these as nested columns and validating they have the same size (which I do not want it to do). I have found no way to disable this behaviour and it is very difficult for me to change these channel names programatically (we ingest them using the s3 table function).
is there some feature I'm perhaps missing or a suggested way to get around this limitation. if not, how hard do you anticipate it being to add such a setting myself

### Which ClickHouse versions are affected?

24.x, 25.x at least

### How to reproduce

ingest a parquet with two array columns (a.x, a.y) where they have different sized arrays

### Expected behavior

a setting to control the nesting inference

### Error message and/or stacktrace

(Array columns) have different array sizes. (SIZES_OF_ARRAYS_DONT_MATCH)

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with a Parquet file containing array columns named a.x and a.y whose arrays have different lengths, using the s3 table function. Trace how nested-column inference and array-size validation are selected. Done means a setting controls that inference and the mismatched arrays can be ingested when nesting is disabled, with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.