Avoid too many remote read when resolve (async commit) lock failed
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
In a product env, TiFlash meet lots of lock error and make lots of remote read ...
"CheckSecondaryLocks error: Exception: CheckSecondaryLocks receives a non-async-commit lock"
"CheckSecondaryLocks error: Exception: server busy"
"resolve txn failed: Exception: resolveAsyncLock failed"
... Finally, those remote reads consume all threads resource and make TiFlash crash.
After the resolve lock fails in one TiFlash node within the backoff retry, even if the remote read is used to resolve lock on another tiflash node, the other tiflash nodes will still fail with a high probability.
Instead of generating a remote read and causing the tiflash process to crash, we can terminate the SQL quickly by throwing an error directly to TiDB when a lock-related error is encountered during the resolve lock phase. This could avoid making TiFlash crashes when meeting locks.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading dbms/src/Storages/Transaction/LearnerRead.cpp around lines 67-74 and dbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp around lines 214-219. Trace how lock-related errors during async-commit resolution lead to remote reads and review the reported failure messages. Done means the SQL terminates with an error instead of generating further remote reads when lock resolution fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100