ClickHouse / ClickHouse/clickhouse-go

Memory leak when write data with http in EncodeColumn

Open
#1,637 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
3.3k
Forks
684
Avg merge
2d 3h
Merged PRs (30d)
14

Description

## Details
When I use the http protocol to write data in batches, I found that there was a serious memory leak problem, and the following is a profile analysis:

Image

and:

Image

What confuses me is that the issue only happens on Kylin v10 aarch64, on this system it works fine with the TCP protocol and works fine on a x86_64 server with http protocol.

I tried to consult deepseek, and deepseek's answer was that:
- the weak memory order of the arm architecture causes go's GC to have significantly higher latency than x86 when reclaiming temporary buffers in ColStr.EncodeXolumn
- Huawei ARM Server has compatibility issues with the memory allocator of go with the HugePages mechanism enabled by default, which exacerbates memory fragmentation

### Environment
* [x] `clickhouse-go` version: v2.21.0
* [x] Interface: ClickHouse API
* [x] Go version: 1.21
* [x] Operating system: KylinV10 aarch64
* [x] ClickHouse version: 23.8.9.55
* [x] Is it a ClickHouse Cloud? No
* [ ] ClickHouse Server non-default settings, if any:
* [ ] `CREATE TABLE` statements for tables involved:
* [ ] Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing batch writes over HTTP with clickhouse-go v2.21.0 on Kylin V10 aarch64, comparing them with TCP and x86_64 HTTP runs. Investigate the EncodeColumn path using the supplied profiles, then verify whether memory remains stable after repeated batches and document findings for the listed Go, ClickHouse, and OS versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.