pingcap / pingcap/tiflash

Slow queries might meet failure with `Can not find disaggregated task`

Open
#10,217 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/enhancement
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Enhancement

In the internal testing disagg-tiflash-workload.ch-partition-hash-run. We meet a query failure as follow

        Error 1105 (HY000): other error for mpp stream: Code: 11004, e.displayText() = DB::Exception: Check snap != nullptr failed: Can not find disaggregated task, task_id=DisTaskId<MPP<gather_id:1, query_ts:1748028447960569885, local_query_id:21
326, server_id:1629, start_ts:458235169457307650,task_id:15>,executor=TableFullScan_81> (from s1008_t5124_12772_3_22445363382982345_177150), e.what() = DB::Exception,

This is because the profiles.default.disagg_task_snapshot_timeout is 5 minute. Under disagg arch, if a slow query that TiFlash compute node can not fetch all mem-tables from the TiFlash write node in 5 minutes, then the query will meet this error and fail.

https://github.com/pingcap/tiflash/blob/b964bf581be2968e1f7ac8b07ee2cf1e7d6eb546/dbms/src/Interpreters/Settings.h#L256
https://github.com/pingcap/tiflash/blob/b964bf581be2968e1f7ac8b07ee2cf1e7d6eb546/dbms/src/Core/Defines.h#L50-L53

Related logging:
0530_endless_7865705.tar.gz

Use tar xvf 0530_endless_7865705.tar.gz for decompress

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dbms/src/Interpreters/Settings.h at the disagg_task_snapshot_timeout setting and dbms/src/Core/Defines.h at the related timeout definition. Review the attached workload log and the disaggregated task snapshot handling to understand why a slow query loses its task. Done means slow queries no longer fail solely because fetching mem-tables exceeds the current timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.