pingcap / pingcap/tidb

lightning's TotalMemoryConsume is not accurate

Open
#53,858 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

only record the significant members

# local engine

## Write step

### local writer

- writeBatch when !isKVSorted

https://github.com/pingcap/tidb/blob/0b5407136bf4953e0de6347069aed7c67bc7455a/pkg/lightning/backend/local/local.go#L1638-L1642

- kvBuffer to save the payload in memory

https://github.com/pingcap/tidb/blob/0b5407136bf4953e0de6347069aed7c67bc7455a/pkg/lightning/backend/local/engine.go#L1187-L1189

## duplicate detection feature

WIP

## Import step

### worker

- KVWriteBatchSize

https://github.com/pingcap/tidb/blob/d3dd2545d3450905780956724c9c27ca696f8fce/pkg/lightning/backend/local/region_job.go#L393-L394

- bufferPool to save the payload in memory

https://github.com/pingcap/tidb/blob/d3dd2545d3450905780956724c9c27ca696f8fce/pkg/lightning/backend/local/region_job.go#L369

- gRPC internal (encoding the message)

- network stack

# externl engine

WIP

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.