test: EventTestRunner.throwException times out during cancellation
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
1. Minimal reproduce step
This was found while validating the TiFlash Jenkins job migration in PingCAP-QE/ci#5062.
- Run TiFlash pull unit tests for pingcap/tiflash#10981, commit
e8b2ad95dd6a58f079ac6b086b8aa5a183c794e3. - Run
gtests_dbms, or specificallyEventTestRunner.throwException, on an 8-vCPU Jenkins worker. - Observe the test timeout while the test is waiting for pipeline tasks to drain.
Relevant test source:
2. What did you expect to see?
EventTestRunner.throwException should finish successfully after the deliberate exception is propagated and the executor is cancelled. The test should not depend on the worker having a particular CPU count.
3. What did you see instead?
The test failed after waiting 15 seconds for active tasks to drain:
[FAILED] EventTestRunner.throwException
DB::Exception: error with timeout
FAILED TESTS (1/3430):
/tiflash/gtests_dbms EventTestRunner.throwException
The failing staging build is pull_unit_test #12, and the migration verifier result is shown here.
The same TiFlash commit passed the original Jenkins job in pull_unit_test #234. The two logs show different worker sizes:
- staging worker: 8 online CPUs
- original worker: 16 online CPUs
Both jobs requested 6 CPUs and used the same TiFlash builder image. This is evidence that worker scheduling/resources expose the problem, but it does not prove that CPU count alone is the root cause.
Root cause analysis
The test creates a default PipelineExecutorContext, whose query_id is empty:
It then schedules 100 DeadLoopEvent instances, each creating 10 tasks that continuously return RUNNING. When ThrowExceptionEvent raises the deliberate exception, PipelineExecutorContext::cancel() only forwards cancellation to TaskScheduler when query_id is non-empty:
Because this test context has an empty query ID, the scheduler cancellation path is skipped. The dead-loop tasks remain active, active_ref_count does not reach zero, and the test's 15-second waitFor() timeout is triggered:
4. What is your TiFlash version?
This is a unit-test issue in the TiFlash source at commit e8b2ad95dd6a58f079ac6b086b8aa5a183c794e3, from pingcap/tiflash#10981. It was observed in the TiFlash builder image ghcr.io/pingcap-qe/cd/builders/tiflash:v2025.4.15-rocky8-llvm-17.0.6-v2.
Proposed fix
The preferred fix is to make the test use a non-empty query ID, so cancellation is routed through the scheduler and reaches the tasks created by this executor context. For example:
- PipelineExecutorContext exec_context;
+ PipelineExecutorContext exec_context{"event-test", "", nullptr};
Please verify that all tasks created by the test inherit this query ID and that the test reliably drains on both 8-vCPU and 16-vCPU workers.
In addition, please consider:
- Adding a regression assertion for cancellation and task draining when an exception is raised.
- Reviewing whether the unit-test cancellation path should support empty query IDs, and adding coverage if that behavior is intended.
- Reducing the number of infinite
DeadLoopTaskinstances or otherwise making the test less scheduling-sensitive. Simply increasing the timeout would only mask the issue. - Treating a worker with at least 16 vCPUs as an infrastructure mitigation, not as the code fix.
A related but distinct production cancellation issue is #11059.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in dbms/src/Flash/Pipeline/Schedule/Events/tests/gtest_event.cpp, especially the EventTestRunner setup, throwException test, and wait() helper. Read PipelineExecutorContext::cancel() and run EventTestRunner.throwException or gtests_dbms to confirm the timeout. Done means the deliberate exception cancels and drains the test tasks reliably on both 8-vCPU and 16-vCPU workers, with regression coverage if needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100