ClickHouse / ClickHouse/clickhouse-go
High memory usage when handling Clickhouse Map
- Dominant language
- Go
- Stars
- 3.3k
- Forks
- 680
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 14
Description
When trying to add the support of Clickhouse Map type in our sinker program(kafka to CH), we noticed much higher CPU and memory usage than before. It does make sense that parsing the map would take time and could result in the CPU usage go higher, but memory usage go about 10 times higher than before is little bit suprising.
We are using the [OrderedMap example ](https://github.com/ClickHouse/clickhouse-go/blob/b92bc86861690ce4eb90be4f6c1918e31701bb40/tests/map_test.go) to construct rows for further writing.
here every row we must construct a map (or even recursive map), we believe that's what caused the high memory usage.
Considering clickhouse supports [Convert Tuple to Map Type](https://clickhouse.com/docs/en/sql-reference/data-types/map#convert-tuple-to-map-type),instead of constucting a map every time, how about constructing two slices called "keys" and "values" with correct mapping? Just think this may be helpful with memory, please share your thoughts.
Contributor guide
Research direction
Start with the OrderedMap example in tests/map_test.go and reproduce the reported CPU and memory behavior for ClickHouse Map values, including recursive maps. Compare it with the pre-Map behavior and inspect the relevant map-writing path. Done means the cause is identified, memory usage is addressed or documented with evidence, and the related tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100