lightning: The CPU is not being effectively utilized when importing compressed files
Open
component/lightning
type/enhancement
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
When importing compressed files, the CPU is not being effectively utilized, resulting in low usage.
The CSV files were compressed and split into 100MB chunks. Even so, importing them from the same region's S3 into the TiDB cluster still took 12 hours to complete the import of 4TiB of data. The Lightning node is configured with 36 cores and 72GB of RAM, but the CPU utilization was only around 750%. It is necessary to increase CPU utilization to accelerate the import process.
Contributor guide
Assessment
This issue has not been assessed yet.