ClickHouse / ClickHouse/clickhouse-go
Memory leak when write data with http in EncodeColumn
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.3k
- Forks
- 684
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 14
Description
## Details
When I use the http protocol to write data in batches, I found that there was a serious memory leak problem, and the following is a profile analysis:
and:
What confuses me is that the issue only happens on Kylin v10 aarch64, on this system it works fine with the TCP protocol and works fine on a x86_64 server with http protocol.
I tried to consult deepseek, and deepseek's answer was that:
- the weak memory order of the arm architecture causes go's GC to have significantly higher latency than x86 when reclaiming temporary buffers in ColStr.EncodeXolumn
- Huawei ARM Server has compatibility issues with the memory allocator of go with the HugePages mechanism enabled by default, which exacerbates memory fragmentation
### Environment
* [x] `clickhouse-go` version: v2.21.0
* [x] Interface: ClickHouse API
* [x] Go version: 1.21
* [x] Operating system: KylinV10 aarch64
* [x] ClickHouse version: 23.8.9.55
* [x] Is it a ClickHouse Cloud? No
* [ ] ClickHouse Server non-default settings, if any:
* [ ] `CREATE TABLE` statements for tables involved:
* [ ] Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing batch writes over HTTP with clickhouse-go v2.21.0 on Kylin V10 aarch64, comparing them with TCP and x86_64 HTTP runs. Investigate the EncodeColumn path using the supplied profiles, then verify whether memory remains stable after repeated batches and document findings for the listed Go, ClickHouse, and OS versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100