pingcap / pingcap/tidb

Lighting precheck shows incorrect estimate sorted data size

Open
#54,216 3 comments 0 reactions 0 assignees View on GitHub
component/lightning severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

Use lightning to import large enough data (>5TB or more) with phsyical import and precheck enabled.

The issue seems related to **converting int64 to float64** during print:
https://github.com/pingcap/tidb/blob/master/lightning/pkg/importer/precheck_impl.go#L587-L588
```
theResult.Message = fmt.Sprintf("local disk resources are rich, estimate sorted data size %s, local available is %s",
units.BytesSize(float64(estimatedDataSizeWithIndex)), units.BytesSize(float64(localAvailable)))
```

https://github.com/pingcap/tidb/blob/master/lightning/pkg/importer/precheck_impl.go#L179-L180
```
theResult.Message += fmt.Sprintf("TiKV requires more storage space. Estimated required size: %s. Actual size: %s.",
units.BytesSize(float64(tikvSourceSize)), units.BytesSize(float64(tikvAvail)))
```

Consider the imported data size is small, it could also relates to tikvSourceSize calculation incorrect and hence trigger such issue.

### 2. What did you expect to see? (Required)
The precheck log should show correct number for estimated data source size.

### 3. What did you see instead (Required)
```
local disk resources are rich, estimate sorted data size -3.074e+18B, local available is xxx(~700GB)
TiKV requires more storage space. Estimated required size: 8EiB. Actual size: xxx(~5TB).
```

### 4. What is your TiDB version? (Required)
v7.5.1

Contributor guide

Open the contributing guide

Research direction

Start in lightning/pkg/importer/precheck_impl.go at the referenced lines around 179-180 and 587-588, then reproduce with a physical import larger than 5 TB and precheck enabled. Check the estimated size calculations and printed values; done means the precheck log reports the correct estimated data source and sorted data sizes instead of negative or 8 EiB values.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.