lightning's TotalMemoryConsume is not accurate
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
only record the significant members
# local engine
## Write step
### local writer
- writeBatch when !isKVSorted
https://github.com/pingcap/tidb/blob/0b5407136bf4953e0de6347069aed7c67bc7455a/pkg/lightning/backend/local/local.go#L1638-L1642
- kvBuffer to save the payload in memory
https://github.com/pingcap/tidb/blob/0b5407136bf4953e0de6347069aed7c67bc7455a/pkg/lightning/backend/local/engine.go#L1187-L1189
## duplicate detection feature
WIP
## Import step
### worker
- KVWriteBatchSize
https://github.com/pingcap/tidb/blob/d3dd2545d3450905780956724c9c27ca696f8fce/pkg/lightning/backend/local/region_job.go#L393-L394
- bufferPool to save the payload in memory
https://github.com/pingcap/tidb/blob/d3dd2545d3450905780956724c9c27ca696f8fce/pkg/lightning/backend/local/region_job.go#L369
- gRPC internal (encoding the message)
- network stack
# externl engine
WIP
Contributor guide
Assessment
This issue has not been assessed yet.