ClickHouse / ClickHouse/clickhouse-go

High memory usage when handling Clickhouse Map

Open
#843 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
3.3k
Forks
680
Avg merge
2d 3h
Merged PRs (30d)
14

Description

When trying to add the support of Clickhouse Map type in our sinker program(kafka to CH), we noticed much higher CPU and memory usage than before. It does make sense that parsing the map would take time and could result in the CPU usage go higher, but memory usage go about 10 times higher than before is little bit suprising.
We are using the [OrderedMap example ](https://github.com/ClickHouse/clickhouse-go/blob/b92bc86861690ce4eb90be4f6c1918e31701bb40/tests/map_test.go) to construct rows for further writing.

here every row we must construct a map (or even recursive map), we believe that's what caused the high memory usage.

Considering clickhouse supports [Convert Tuple to Map Type](https://clickhouse.com/docs/en/sql-reference/data-types/map#convert-tuple-to-map-type),instead of constucting a map every time, how about constructing two slices called "keys" and "values" with correct mapping? Just think this may be helpful with memory, please share your thoughts.

Contributor guide

Open the contributing guide

Research direction

Start with the OrderedMap example in tests/map_test.go and reproduce the reported CPU and memory behavior for ClickHouse Map values, including recursive maps. Compare it with the pre-Map behavior and inspect the relevant map-writing path. Done means the cause is identified, memory usage is addressed or documented with evidence, and the related tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.