pingcap / pingcap/tidb

Lightning: global metadata manager for single mode import

Open
#35,022 1 comment 0 reactions 0 assignees View on GitHub
component/lightning type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
https://github.com/pingcap/tidb/pull/34288#discussion_r882542718

As is mentioned in this link, Lightning only maintains metadata globally for parallel import, where multiple Lightning instances are involved. When importing in single mode, however, `singleMetaManger` merely maintains nothing (or few things) and most metadata are stored in `TableRestore`. It's better to have a global metadata manager because:
1. We don't need to care much about parallel import when developing a new feature. All the information that are potentially needed to reconcile with other instances are well abstracted from `TableRestore` and manipulated by the metadata manager interface.
2. A good and globally unified metadata manager can prominently reduce branches from current codes, where lots of them are concerning about the special case of parallel import, making the logic unfathomable and not concise.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.