global sort writer uses more memory than expected
Open
type/enhancement
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
I have set 64MB memory in a global sort writer test
https://github.com/pingcap/tidb/blob/c82e19c0c29a18ea77933caa1529b7eac7a63824/br/pkg/lightning/backend/external/bench_test.go#L198
https://github.com/pingcap/tidb/blob/c82e19c0c29a18ea77933caa1529b7eac7a63824/br/pkg/lightning/backend/external/bench_test.go#L144-L150
However, when check the heap profile, I see we forget to take `kvLocations` into consideration
https://github.com/pingcap/tidb/blob/c82e19c0c29a18ea77933caa1529b7eac7a63824/br/pkg/lightning/backend/external/writer.go#L346
While `kvBuffer` allocated exactly 64MB memory, `kvLocations` uses 30.47MB memory when `key size: 20, value size: 10`
Contributor guide
Assessment
This issue has not been assessed yet.