pingcap / pingcap/tiflash

Large sparse table cause apply snapshot slow

Open
#8,000 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/enhancement
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Enhancement

A sparse table with average row size~15.17 bytes.
Because we set region-split-keys = 50000000 and region-split-size = "500MiB" on serverless tier, its regions snapshot contains more than 54,000,000 rows.
Decoding the key-values of the snapshot takes 220 seconds and ingest it into Delta-Tree cost 17 seconds.

The ingest phase is executed by only 1 thread, these slow ingest become the bottleneck when we adding TiFlash replica for a large amount of Regions cause regions apply snapshot timeout.

After changing following configs, the ingest cost down to less than 4 seconds.

[profiles]
[profiles.default]
dt_segment_limit_rows = 5000000
dt_segment_force_merge_delta_rows = 671088640
dt_segment_stop_write_delta_rows = 1342177280

We need more tests to ensure whether it can become the default config values in the master branch.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the profile settings for dt_segment_limit_rows, dt_segment_force_merge_delta_rows, and dt_segment_stop_write_delta_rows, then find tests covering sparse-table snapshot ingest. Compare the current defaults with the reported values and measure ingest time for large snapshots. Done means tests justify whether these values can become the master-branch defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.