ClickHouse / ClickHouse/ClickHouse

Nondeterministic functions (`rand`, `generateUUIDv4`, `generateSnowflakeID`, ...) return one value per dictionary key when applied to a `LowCardinality` column

Open
#117,466 0 comments 0 reactions 1 assignee Claimed by @KochetovNicolai View on GitHub
bug comp-datatype-wrapper comp-functions
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Describe what's wrong**

Nondeterministic functions applied to a `LowCardinality` column return one shared value per dictionary key instead of an independent value per row, at pure default settings. Every function that declares `isDeterministicInScopeOfQuery() == false` and takes a column argument is affected; verified for `generateUUIDv4`, `generateUUIDv7`, `generateSnowflakeID`, `rand`, `rand64`, `randomString` and `randCanonical`. The wrong values are silently persisted by `INSERT ... SELECT`, so unique identifiers generated this way are duplicated across rows.

The column argument of these functions exists precisely so that callers can defeat common subexpression elimination and get an independent value per row, so applying them to a table column is the intended usage. Passing a `LowCardinality` column — the natural type for the kind of column such expressions are keyed to — silently breaks it.

Root cause: `IFunctionOverloadResolver::getReturnType` chooses a `LowCardinality` result type whenever `canBeExecutedOnLowCardinalityDictionary()` allows it (src/Functions/IFunction.cpp:817), and that property defaults to `true` for every function (src/Functions/IFunction.h:518) with no determinism check. The execution path then replaces the `LowCardinality` argument by its dictionary (`replaceLowCardinalityColumnByNestedAndGetDictionaryIndexes`, src/Functions/IFunction.cpp:502), executes the function once per dictionary key, and fans the per-key results back out to all rows through the index column (`res_indexes->index(*indexes, 0)`, src/Functions/IFunction.cpp:531). That transformation is only sound when the function returns the same value for the same argument.

This is the `LowCardinality` sibling of #117224 (constant arguments, fixed by #117358) and #117209 (sparse columns, fix in #117461); neither fix covers this path.

**Does it reproduce on the most recent release?**

Yes, reproduced on master 26.9.1.460 (and on 26.9.1.1).

**How to reproduce**

```sql
CREATE TABLE t_lc (k LowCardinality(String)) ENGINE = MergeTree ORDER BY tuple();
INSERT INTO t_lc SELECT toString(number % 10) FROM numbers(1000);

SELECT uniqExact(generateUUIDv4(k)) FROM t_lc; -- 10, expected 1000
SELECT uniqExact(generateSnowflakeID(k)) FROM t_lc; -- 10, expected 1000
SELECT uniqExact(rand(k)) FROM t_lc; -- 10, expected 1000
SELECT uniqExact(randomString(16, k)) FROM t_lc; -- 10, expected 1000

-- the duplicates are persisted
CREATE TABLE ids (u UUID) ENGINE = MergeTree ORDER BY tuple();
INSERT INTO ids SELECT generateUUIDv4(k) FROM t_lc;
SELECT count(), uniqExact(u) FROM ids; -- 1000, 10

-- control: the same data as plain String behaves correctly
CREATE TABLE t_plain (k String) ENGINE = MergeTree ORDER BY tuple();
INSERT INTO t_plain SELECT toString(number % 10) FROM numbers(1000);
SELECT uniqExact(generateUUIDv4(k)) FROM t_plain; -- 1000
```

All settings are at their defaults.

**Expected behavior**

A nondeterministic function evaluates once per row regardless of the argument column's serialization, so every `uniqExact` above returns 1000 and the `ids` table holds 1000 distinct values. The dictionary fast path should be reserved for functions that are deterministic in the scope of the query, e.g. by consulting `isDeterministicInScopeOfQuery` where `canBeExecutedOnLowCardinalityDictionary` is applied, the same way #117461 gates the sparse fast path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.