pingcap / pingcap/tiflash

[potential bug] TiFlash replica not available for about 10 minutes with error `tikv:9006`.

Open
#1,447 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

During running schrodinger/sqllogic test, I met the following problem:

The test process

  1. created a table,
  2. add a tiflash replica,
  3. wait for the replica available, but it not available for about 10 minutes.

Note that before this operation, there was a heavy load in this tidb cluster produced by other threads of the sqllogic test.
And when it was waiting, all of the load was hanging.

tiflash_cluster_manager.log

Exception: can not get tiflash replica info from tidb: [('127.0.0.1:8539', Exception('unexpected status code 400 from http://127.0.0.1:8539/tiflash/replica'))]
2021-02-22 13:39:09,528 <ERROR> root: can not get tiflash replica info from tidb: [('127.0.0.1:8539', Exception('unexpected status code 400 from http://127.0.0.1:8539/tiflash/replica'))]
Traceback (most recent call last):
  File "flash_cluster_manager.py", line 286, in main
  File "flash_cluster_manager.py", line 129, in __init__
  File "flash_cluster_manager.py", line 29, in wrap_func
  File "flash_cluster_manager.py", line 238, in table_update
  File "tidb_tools.py", line 42, in db_flash_replica

http://h81:8539/tiflash/replica
(h81 is the host of the cluster, 8539 is tidb status port.)

[tikv:9006]GC life time is shorter than transaction duration, transaction starts at 2021-02-22 13:21:08.223 +0800 CST, GC safe point is 2021-02-22 13:21:08.923 +0800 CST

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the schrodinger/sqllogic sequence under heavy cluster load and inspect the tiflash_cluster_manager.py and tidb_tools.py call paths named in the traceback. Check the /tiflash/replica response alongside the tikv:9006 GC lifetime error. Done means identifying and resolving the cause of the delayed replica availability and hanging test load.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.