tikv / tikv/pd

scheduler: Region score may underreact to physical disk skew caused by heterogeneous data compressibility

Open
#11,036 1 comment 0 reactions 0 assignees View on GitHub
contribution needs-type
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

Related to #3577.

### Background

We observed a production case on a v8.5.6-based cluster where different application-data compressibility caused significant physical disk skew between TiKV Stores.

TiKV engineering double-checked the behavior. With identical RocksDB compression settings, the compression-ratio difference is caused by the compressibility of the written data rather than a TiKV compression defect. The affected Store accumulated a substantially larger physical default CF L6 SST footprint.

At the same observation point:

| | Store A | Store B |
|---|---:|---:|
| Physical Store used | ~2.4 TB | ~1.8 TB |
| Smoothed available space (A) | ~726 GiB | ~1,317 GiB |
| Logical Region size (R) | ~6.95M MiB | ~7.75M MiB |
| Region score | ~8.90M | ~9.00M |

Image

Although Store A had substantially higher physical disk usage, its smaller logical Region size offset the higher disk-space multiplier. The two Region scores therefore remained close and PD did not generate strong outbound balancing pressure.

### Current Region score behavior

For the normal-space branch of Region score v2:

```
F = max(50, C * (1 - low-space-ratio))

score = [1 + 256 * (ln(C) - ln(A - F + 1))
/ (C - A + F - 1)] * R
```

Here, `C` is Store capacity, `A` is smoothed available space, and `R` is logical Region size.

The disk has a nominal capacity of 3.2 TB, reported to PD as approximately 2,980 GiB. With the default `low-space-ratio=0.8`:

```
F ~= 2,980 GiB * (1 - 0.8) ~= 596 GiB
```

PD applies gradual disk-space pressure above this boundary, but the much stronger low-space penalty is used only when smoothed available space approaches or falls below approximately 596 GiB.

The observed scores can be approximated as:

```
Store A: 6.95M * 1.281 ~= 8.90M
Store B: 7.75M * 1.161 ~= 9.00M
```

This is mathematically consistent with the current formula, but it can delay correction of substantial physical disk skew caused by heterogeneous data compressibility.

### Expected improvement

PD should apply earlier and smoother outbound balancing pressure when physical available-space divergence becomes large, before the affected Store reaches the emergency low-space boundary.

This does not necessarily mean scheduling directly by physical SST bytes, because SST files can span multiple Regions and PD does not know exactly how many physical bytes moving a Region will release. Possible directions include:

- adjusting the normal-space multiplier curve so that large available-space divergence has a stronger effect;
- incorporating a smoothed and capped Store-level physical amplification estimate; or
- adding another safeguard before the hard low-space branch.

The goal is to prevent a Store with significantly higher physical disk usage from retaining a Region score close to Stores with much more available space solely because logical Region size offsets the disk-space multiplier.

### Version

Observed with PD Region score v2 behavior in a v8.5.6-based deployment.

Contributor guide

Open the contributing guide

Research direction

Start by tracing PD's Region score v2 normal-space branch and reviewing related issue #3577. The issue names no file, test, or entry point, so locate the score implementation and its existing coverage before choosing among the proposed approaches. Done means a defined change applies earlier, smoother pressure to physical disk skew without relying only on the emergency low-space branch, with regression evidence for the reported scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.