pingcap / pingcap/tiflash

More debug interfaces for troubleshooting

Open
#2,800 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/feature-request
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

When doing troubleshooting for https://github.com/pingcap/tics/issues/2756, these tools are very helpful:

  • A tool for comparing the result from TiKV to TiFlash, finding out the rowkey(s)/Region(s) that is inconsistent
    • @JinheLin has implemented a prototype for it, but it only supports finding one different rowkey for non-clustered table
    • Locating by select count(*) from <table> where <min-key> <= _tidb_rowid and _tidb_rowid < <max-key>;
    • Setting set tidb_allow_batch_cop =0; and set tidb_allow_mpp = 0; would be helpful for locating the inconsistent rows
    • Need retry for checking multiple replicas
    • Locating the Region instead of rowkey would be greatly helpful
  • TiFlash provides a debug interface for showing the overlapped between Regions and Segments
  • TiFlash provides a debug interface for showing the details of stable and delta layer
    • Stable: the DTFile(s)
    • Delta: the type, id of DeltaPacks

These tools maybe helpful for troubleshooting:

  • https://github.com/disksing/mok encoding table id & rowkey into Region key / decoding Region key into table id & rowkey
  • pd-ctl doc
    • Find out all TiFlash stores: store --jq='.stores[].store | select(.labels | length>0 and contains([{"key":"engine","value":"tiflash"}])) | { id, address, state_name}'
    • Locate Region by key: region key <key>
    • Remove the Region peer: operator add remove-peer <region-id> <store-id>
  • Dump the Regions of a table inside one TiFlash node:./bin/tiflash/tiflash client --host <tiflash-ip> --port <tiflash-tcp-port> --query "DBGInvoke dump_all_region(<table-id>)"

Need further investigation:

  • Can PD locate the Region IDs by a key range?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the prototype mentioned in the issue, the linked troubleshooting issue 2756, and the pd-ctl commands plus the TiFlash client DBGInvoke dump_all_region entry point. Investigate how to compare TiKV and TiFlash results, locate Regions, and expose Region/Segment and stable/delta details. Done should mean the needed troubleshooting interfaces support the listed replica, row, Region, and storage-layer cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
databases, distributed-systems, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.