ClickHouse / ClickHouse/clickhouse-go
Map(K, V) with duplicate keys: values silently collapse when the map is nested (Array(Map), Map(K, Map))
- Dominant language
- Go
- Stars
- 3.3k
- Forks
- 680
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 14
Description
## Description
ClickHouse `Map(K, V)` is [documented](https://clickhouse.com/docs/sql-reference/data-types/map) as *not* being a set of unique-by-key pairs: "a map can contain two elements with the same key". The native protocol faithfully delivers every key/value pair (the keys and values sub-columns plus offsets), but `lib/column/map.go` materialises rows into a Go `map`, so duplicate keys silently collapse to the last value.
For a **top-level** `Map` column there is a working escape hatch: scanning into `*orderedmap.Map[K, V]` goes through `Map.orderedRow` and preserves all pairs (`orderedmap.Map` is slice-backed and its `Put` appends). That path is fine.
The problem is **nested** maps — `Array(Map(K, V))`, `Map(K, Map(K, V))`, `Tuple(..., Map(K, V))`, etc. There the inner `Map` column is materialised by the parent column via `Interface.Row(i, ptr)`, which calls `Map.row` → `reflect.Value.SetMapIndex`. `orderedRow` is only reachable from `Map.ScanRow` on the outermost column, so there is **no** way for a caller to get all the pairs of a nested map: data present on the wire is dropped before the caller can see it.
## ClickHouse server version
`26.7.3` (the testcontainers image used by the integration suite; `clickhouse/clickhouse-server:latest`).
## Reproduction
```go
package tests
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"github.com/ClickHouse/clickhouse-go/v2"
"github.com/ClickHouse/clickhouse-go/v2/lib/column/orderedmap"
)
func TestMapDuplicateKeys(t *testing.T) {
conn, err := GetNativeConnection(t, clickhouse.Native, nil, nil, nil)
require.NoError(t, err)
t.Cleanup(func() { conn.Close() })
ctx := context.Background()
// The server really does send two pairs.
var n uint64
require.NoError(t, conn.QueryRow(ctx, "SELECT length(map('key', 'X', 'key', 'Y'))").Scan(&n))
require.Equal(t, uint64(2), n) // passes
// Top level, orderedmap destination: OK, both pairs preserved.
om := orderedmap.Map[string, string]{}
require.NoError(t, conn.QueryRow(ctx, "SELECT map('key', 'X', 'key', 'Y')").Scan(&om))
require.Len(t, om, 2) // passes -> [{key X} {key Y}]
// Nested in an Array: one pair lost, no alternative destination available.
var am []map[string]string
require.NoError(t, conn.QueryRow(ctx, "SELECT [map('key', 'X', 'key', 'Y')]").Scan(&am))
require.Len(t, am[0], 2) // FAILS: got map[key:Y] (len 1)
// Nested in a Map: same.
var mm map[string]map[string]string
require.NoError(t, conn.QueryRow(ctx, "SELECT map('outer', map('key', 'X', 'key', 'Y'))").Scan(&mm))
require.Len(t, mm["outer"], 2) // FAILS: got map[outer:map[key:Y]] (inner len 1)
}
```
Observed output from the run:
```
plain map => map[key:Y] (len=1) # top level, map destination (expected, documented-ish)
orderedmap => [{key X} {key Y}] (2) # top level, escape hatch works
array of map => [map[key:Y]] # nested: X lost, no escape hatch
map in map => map[outer:map[key:Y]] # nested: X lost, no escape hatch
server map length => 2
```
Expected: a caller should be able to retrieve both `key=X` and `key=Y` for a nested map, as they can today for a top-level one.
## Suggested fix
- `lib/column/map.go:304-333` (`Map.row`) is where the collapse happens, via `value.SetMapIndex`. It is the only representation available to parent columns, since `Array`/`Map`/`Tuple` reach their children through `Interface.Row`.
- Keeping `Map.row` as the default `Row` representation is reasonable for backwards compatibility, but nested maps need some opt-in path to the ordered/duplicate-preserving representation — e.g. plumbing a per-query or per-column preference so that a nested `Map` materialises as an `orderedmap.Map` (its `ScanType` would change accordingly), or letting `Array.ScanRow`/`Map.ScanRow` recurse into children with the caller's requested element type instead of always going through `Row`.
- Note that `Map.scanType` is also what `Rows.ColumnTypes()`/`ScanType()` advertise, so any change there needs to stay opt-in.
## Link
Reported for clickhouse-java as https://github.com/ClickHouse/clickhouse-java/issues/3047 (same root cause: the reader merges pairs into a unique-key map). Tracking: https://github.com/ClickHouse/integrations-ai-playground/issues/389
Contributor guide
Research direction
Start with lib/column/map.go:304-333 and trace Map.row through Interface.Row when Array, Map, or Tuple materialises nested children. Run the TestMapDuplicateKeys reproduction against the integration suite, then determine an opt-in way for nested maps to preserve duplicate pairs without changing the default Row representation or scan type. Done means both nested examples retain two pairs while existing top-level map behavior remains compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100