More debug interfaces for troubleshooting
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
When doing troubleshooting for https://github.com/pingcap/tics/issues/2756, these tools are very helpful:
- A tool for comparing the result from TiKV to TiFlash, finding out the rowkey(s)/Region(s) that is inconsistent
- @JinheLin has implemented a prototype for it, but it only supports finding one different rowkey for non-clustered table
- Locating by
select count(*) from <table> where <min-key> <= _tidb_rowid and _tidb_rowid < <max-key>; - Setting
set tidb_allow_batch_cop =0;andset tidb_allow_mpp = 0;would be helpful for locating the inconsistent rows - Need retry for checking multiple replicas
- Locating the Region instead of rowkey would be greatly helpful
- TiFlash provides a debug interface for showing the overlapped between Regions and Segments
- TiFlash provides a debug interface for showing the details of stable and delta layer
- Stable: the DTFile(s)
- Delta: the type, id of DeltaPacks
These tools maybe helpful for troubleshooting:
- https://github.com/disksing/mok encoding table id & rowkey into Region key / decoding Region key into table id & rowkey
- pd-ctl doc
- Find out all TiFlash stores:
store --jq='.stores[].store | select(.labels | length>0 and contains([{"key":"engine","value":"tiflash"}])) | { id, address, state_name}' - Locate Region by key:
region key <key> - Remove the Region peer:
operator add remove-peer <region-id> <store-id>
- Find out all TiFlash stores:
- Dump the Regions of a table inside one TiFlash node:
./bin/tiflash/tiflash client --host <tiflash-ip> --port <tiflash-tcp-port> --query "DBGInvoke dump_all_region(<table-id>)"
Need further investigation:
- Can PD locate the Region IDs by a key range?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the prototype mentioned in the issue, the linked troubleshooting issue 2756, and the pd-ctl commands plus the TiFlash client DBGInvoke dump_all_region entry point. Investigate how to compare TiKV and TiFlash results, locate Regions, and expose Region/Segment and stable/delta details. Done should mean the needed troubleshooting interfaces support the listed replica, row, Region, and storage-layer cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- databases, distributed-systems, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100