ClickHouse / ClickHouse/clickhouse-go
Improvement: Too much memory consumption on Native format
- Dominant language
- Go
- Stars
- 3.3k
- Forks
- 680
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 14
Description
## Observed
This is observed by internal observability team when trying to copy system tables from bunch of ClickHouse instances into another ClickHouse instance using Native format.
## Expected behaviour
The expected behavior is to be memory efficient, given we move data from one clickhouse into another in native clickhouse format (no need for additional serialization/deserialization)
## Details
@RoryCrispin @velom reported this issue few weeks ago. IIRC, they [use specific tag of clickhouse-go client](https://github.com/ClickHouse/clickhouse-go/pull/1564)
### Environment
* [ ] `clickhouse-go` version:
* [ ] Interface: ClickHouse API / `database/sql` compatible driver
* [ ] Go version:
* [ ] Operating system:
* [ ] ClickHouse version:
* [ ] Is it a ClickHouse Cloud?
* [ ] ClickHouse Server non-default settings, if any:
* [ ] `CREATE TABLE` statements for tables involved:
* [ ] Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
Contributor guide
Research direction
Start by reviewing the linked clickhouse-go PR #1564 and filling in the missing environment, version, table, and sample-data details. Compare memory use while copying ClickHouse system tables in Native format; done means the transfer is demonstrably more memory-efficient without unnecessary serialization or deserialization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100