ClickHouse / ClickHouse/clickhouse-go

LowCardinality(Float32/Float64): -0 is written as +0, and NaN rows are written as the dictionary default

Open
#2,007 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3.3k
Forks
680
Avg merge
2d 3h
Merged PRs (30d)
14

Description

`LowCardinality` builds its block dictionary by deduplicating on the Go value with a `map[any]int`. For float inner types, Go value equality does not agree with the wire encoding, and two rows that must stay distinct get merged into one dictionary entry. The values reach the server changed.

Two separate defects come out of the same lookup, both verified below. Neither needs a server to reproduce.

## 1. `-0` is written as `+0`

Go says `float32(+0) == float32(-0)`, so the second row finds the first one's dictionary entry. Their wire encodings differ (`0x00000000` vs `0x80000000`), so the sign is dropped.

## 2. `NaN` rows are written as the dictionary default

Go says `NaN != NaN`, so the lookup at [`lowcardinality.go:147`](https://github.com/ClickHouse/clickhouse-go/blob/main/lib/column/lowcardinality.go#L147) misses and a new entry is appended. But [line 153](https://github.com/ClickHouse/clickhouse-go/blob/main/lib/column/lowcardinality.go#L153) then reads `col.append.index[v]` a *second* time, which misses again and yields the zero value `0`. Every `NaN` row is keyed to the reserved slot 0 and is written as `0.0` (or as `NULL` for a nullable inner), while the dictionary still grows one unreachable entry per `NaN` row.

## Reproduction

`go.mod` requires `github.com/ClickHouse/clickhouse-go/v2 v2.48.0`.

```go
package main

import (
"fmt"
"math"
"time"

"github.com/ClickHouse/ch-go/proto"
"github.com/ClickHouse/clickhouse-go/v2/lib/column"
)

var sc = &column.ServerContext{Timezone: time.UTC}

func dump(label, chType string, values []any) {
col, err := column.Type(chType).Column("c", sc)
if err != nil {
fmt.Printf("%-28s column error: %v\n", label, err)
return
}
for _, v := range values {
if err := col.AppendRow(v); err != nil {
fmt.Printf("%-28s append error: %v\n", label, err)
return
}
}

buf := &proto.Buffer{}
col.Encode(buf)
fmt.Printf("%-28s rows=%d body=%x\n", label, col.Rows(), buf.Buf)
}

func main() {
negZero32 := float32(math.Copysign(0, -1))
negZero64 := math.Copysign(0, -1)

dump("Float32 +0,-0", "LowCardinality(Float32)", []any{float32(0), negZero32})
dump("Float64 +0,-0", "LowCardinality(Float64)", []any{float64(0), negZero64})
dump("Float32 NaN,NaN,1.5", "LowCardinality(Float32)", []any{float32(math.NaN()), float32(math.NaN()), float32(1.5)})
dump("String a,b,a", "LowCardinality(String)", []any{"a", "b", "a"})
dump("plain Float32 +0,-0", "Float32", []any{float32(0), negZero32})
}
```

Output on clickhouse-go v2.48.0, Go 1.26.4, linux/amd64:

```
Float32 +0,-0 rows=2 body=00060000000000000200000000000000000000000000000002000000000000000101
Float64 +0,-0 rows=2 body=000600000000000002000000000000000000000000000000000000000000000002000000000000000101
Float32 NaN,NaN,1.5 rows=3 body=00060000000000000400000000000000000000000000c07f0000c07f0000c03f0300000000000000000003
String a,b,a rows=3 body=0006000000000000030000000000000000016101620300000000000000010201
plain Float32 +0,-0 rows=2 body=0000000000000080
```

Decoded:

**`Float32 +0,-0`** — the dictionary holds `+0` twice and both keys point at index 1. `-0` is not in the output at all.

```
0006000000000000 metadata (key width 0, block-local dictionary)
0200000000000000 dict_size = 2
00000000 dict[0] = +0 (reserved default)
00000000 dict[1] = +0 <- should be -0 (0x80000000)
0200000000000000 keys_count = 2
01 01 both rows -> dict[1]
```

**`Float32 NaN,NaN,1.5`** — two unreachable `NaN` entries, and both `NaN` rows key to slot 0, which holds `+0`.

```
0400000000000000 dict_size = 4
00000000 dict[0] = +0 (reserved default)
0000c07f dict[1] = NaN (unreachable)
0000c07f dict[2] = NaN (unreachable)
0000c03f dict[3] = 1.5
0300000000000000 keys_count = 3
00 00 03 NaN rows -> dict[0] = +0
```

`LowCardinality(String)` is correct, and plain `Float32` writes `0000000000000080`, preserving the sign. The defect is specific to the LowCardinality dictionary.

Both types need `allow_suspicious_low_cardinality_types=1` server-side, so this is not on the common path, but the failure is silent: no error is returned, and the bytes sent are not the values appended. I checked the encoder output rather than a server round-trip.

## Cause

[`lowcardinality.go:147-153`](https://github.com/ClickHouse/clickhouse-go/blob/main/lib/column/lowcardinality.go#L147-L153):

```go
if _, found := col.append.index[v]; !found {
if err := col.index.AppendRow(v); err != nil {
return err
}
col.append.index[v] = col.index.Rows() - 1
}
col.append.keys = append(col.append.keys, col.append.index[v])
```

A dictionary key must identify a value's *encoding*, but `map[any]int` compares the Go value. For floats those two relations disagree in both directions: `==` merges `+0`/`-0`, and it separates every `NaN` from itself.

## Possible fixes

For defect 2, capturing the index instead of re-reading the map is enough to stop `NaN` rows landing on slot 0 (ch-go already does this in `ColLowCardinality.Prepare`), though it leaves one dictionary entry per `NaN` row.

Defect 1 needs the map key to be the encoded form. Keying floats on `math.Float32bits`/`math.Float64bits` would make dictionary equality match wire equality for those types.

Worth a look at `time.Time` while in here: the `Truncate(time.Second)` at [line 144](https://github.com/ClickHouse/clickhouse-go/blob/main/lib/column/lowcardinality.go#L144) is applied to the value that gets appended, so `LowCardinality(DateTime64(N))` looks like it would lose sub-second precision. I have not confirmed that one.

## Also in ch-go

`ColLowCardinality.Prepare` keys `map[T]int` on the value too, so the `-0` merge is there as well:

```
ch-go LowCardinality(Float32) +0,-0 rows=2 body=000600000000000001000000000000000000000002000000000000000000
ch-go Float32 +0,-0 rows=2 body=0000000000000080
```

A one-entry dictionary holding `+0`, with both keys pointing at it. Being generic over `comparable` T, ch-go does not have defect 2. Not filed separately.

Contributor guide

Open the contributing guide

Research direction

Start in lib/column/lowcardinality.go around lines 144-153, then run the supplied encoder reproduction without a server. Add regression coverage for signed zero and NaN in LowCardinality float columns, verifying that encoded values and dictionary keys remain correct; also check the existing String behavior is unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.