[potential bug] TiFlash replica not available for about 10 minutes with error `tikv:9006`.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
During running schrodinger/sqllogic test, I met the following problem:
The test process
- created a table,
- add a tiflash replica,
- wait for the replica available, but it not available for about 10 minutes.
Note that before this operation, there was a heavy load in this tidb cluster produced by other threads of the sqllogic test.
And when it was waiting, all of the load was hanging.
tiflash_cluster_manager.log
Exception: can not get tiflash replica info from tidb: [('127.0.0.1:8539', Exception('unexpected status code 400 from http://127.0.0.1:8539/tiflash/replica'))]
2021-02-22 13:39:09,528 <ERROR> root: can not get tiflash replica info from tidb: [('127.0.0.1:8539', Exception('unexpected status code 400 from http://127.0.0.1:8539/tiflash/replica'))]
Traceback (most recent call last):
File "flash_cluster_manager.py", line 286, in main
File "flash_cluster_manager.py", line 129, in __init__
File "flash_cluster_manager.py", line 29, in wrap_func
File "flash_cluster_manager.py", line 238, in table_update
File "tidb_tools.py", line 42, in db_flash_replica
http://h81:8539/tiflash/replica
(h81 is the host of the cluster, 8539 is tidb status port.)
[tikv:9006]GC life time is shorter than transaction duration, transaction starts at 2021-02-22 13:21:08.223 +0800 CST, GC safe point is 2021-02-22 13:21:08.923 +0800 CST
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the schrodinger/sqllogic sequence under heavy cluster load and inspect the tiflash_cluster_manager.py and tidb_tools.py call paths named in the traceback. Check the /tiflash/replica response alongside the tikv:9006 GC lifetime error. Done means identifying and resolving the cause of the delayed replica availability and hanging test load.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100