Lightning: reduce the duration when precheck parquet file
Open
component/lightning
type/enhancement
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
When the data file is in Parquet format, there is no need to read rows, which is time-consuming and resource-consuming.
https://github.com/pingcap/tidb/blob/cf8a941d109ac3b8a0c7f2be422dcedc9d0032aa/br/pkg/lightning/importer/precheck_impl.go#L1018
If parquet file size is less than 256MB, Lightning will download entire file to local
https://github.com/pingcap/tidb/blob/cf8a941d109ac3b8a0c7f2be422dcedc9d0032aa/br/pkg/lightning/mydump/parquet_parser.go#L125-L135
Contributor guide
Assessment
This issue has not been assessed yet.