Slow queries might meet failure with `Can not find disaggregated task`
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
In the internal testing disagg-tiflash-workload.ch-partition-hash-run. We meet a query failure as follow
Error 1105 (HY000): other error for mpp stream: Code: 11004, e.displayText() = DB::Exception: Check snap != nullptr failed: Can not find disaggregated task, task_id=DisTaskId<MPP<gather_id:1, query_ts:1748028447960569885, local_query_id:21
326, server_id:1629, start_ts:458235169457307650,task_id:15>,executor=TableFullScan_81> (from s1008_t5124_12772_3_22445363382982345_177150), e.what() = DB::Exception,
This is because the profiles.default.disagg_task_snapshot_timeout is 5 minute. Under disagg arch, if a slow query that TiFlash compute node can not fetch all mem-tables from the TiFlash write node in 5 minutes, then the query will meet this error and fail.
https://github.com/pingcap/tiflash/blob/b964bf581be2968e1f7ac8b07ee2cf1e7d6eb546/dbms/src/Interpreters/Settings.h#L256
https://github.com/pingcap/tiflash/blob/b964bf581be2968e1f7ac8b07ee2cf1e7d6eb546/dbms/src/Core/Defines.h#L50-L53
Related logging:
0530_endless_7865705.tar.gz
Use tar xvf 0530_endless_7865705.tar.gz for decompress
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with dbms/src/Interpreters/Settings.h at the disagg_task_snapshot_timeout setting and dbms/src/Core/Defines.h at the related timeout definition. Review the attached workload log and the disaggregated task snapshot handling to understand why a slow query loses its task. Done means slow queries no longer fail solely because fetching mem-tables exceeds the current timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100