ClickHouse / ClickHouse/ClickHouse

Inconsistent behavior working with large arrays

Open
#29,532 1 comment 0 reactions 0 assignees View on GitHub
comp-query-execution usability
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

We have internal limitation to the maximum array size which can be stored inside `Field` structure -- 1000000. This value is reasonable and we shouldn't allow to have bigger arrays in fields. But we have following inconsistencies:
```
:) SELECT range(1000001) -- expected to fail, 1000001 > 1000000

Received exception from server (version 21.11.1):
Code: 128. DB::Exception: Received from localhost:9000. DB::Exception: Array of size 1000001 is too large to be manipulated as single field, maximum size 1000000. (TOO_LARGE_ARRAY_SIZE)
```
But, semantically same query works:
```
:) SELECT groupArray(number) AS n_array -- array is bigger than 1000000
FROM
(
SELECT number
FROM numbers(1000001)
)

┌─n_array────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ [0,1,2,3,4,5,6,7,8,9,⋯│
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```

And again, if we move the array into `WITH` expression query will fail:
```
:) WITH (
SELECT groupArray(number) AS n_array
FROM
(
SELECT number
FROM numbers(1000001)
)
) AS A
SELECT
A

Received exception from server (version 21.11.1):
Code: 128. DB::Exception: Received from localhost:9000. DB::Exception: Array of size 1000001 is too large to be manipulated as single field, maximum size 1000000. (TOO_LARGE_ARRAY_SIZE)
```

The main reason is that we have `Const` column in case of `range(...)` and `WITH` which tries to convert the value to `Field`, but in case №2 we have Array(UInt64). It's not visible from users size:
```

:) SELECT toTypeName(range(1000001))

┌─toTypeName(range(1000001))─┐
│ Array(UInt32) │
└────────────────────────────┘

:) SELECT toTypeName(groupArray(number))
FROM
(
SELECT number
FROM numbers(1000001)
)

┌─toTypeName(groupArray(number))─┐
│ Array(UInt64) │
└────────────────────────────────┘

:) WITH (
SELECT groupArray(number) AS n_array
FROM
(
SELECT number
FROM numbers(1000001)
)
) AS A
SELECT toTypeName(A)

┌─toTypeName(A)─┐
│ Array(UInt64) │
└───────────────┘
```

So we have to make this behavior consistent.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.