pingcap / pingcap/tiflash

Query may hang when remote read use cop stream and bounded queue

Open
#8,177 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/compute severity/moderate type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Suppose the probe side of a join uses remote read and starts to run before the build side. The remote read from probe side may totally occupy the fixed-size thread pool of cop stream. These remote reads from probe side may hang because the queue used for receiving responses is bounded. Then if the build side also needs to do a remote read, this remote read can not proceed because the thread pool is full.
The waiting sequence is

  1. Remote reads from build side are waiting for a slot in the thread pool.
  2. Remote reads from probe side(which occupies the thread pool) are waiting for build phase to complete.

As a result, the query hangs forever.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the cop stream fixed-size thread pool and the bounded response queue involved in remote reads, using the described build-before-probe ordering to reproduce the hang. Done means the query no longer deadlocks when probe-side reads occupy the pool and build-side reads also require remote reads, with a regression test covering that sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.