pingcap / pingcap/tiflash

[tidbx] query report error when injection tiflash wn network partition

Open
#10,994 2 comments 0 reactions 1 assignee View on GitHub

@yongman is already working on this.

Since Jul 21, 2026.

type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

deploy with tiflash cn (2 replicas) + wn (2 replicas)
1, run ch workload with 2000 warehouse
go-tpc ch run -D tpcc --host tc-tidb.xxx -P4000 --warehouses 2000 -T 32 --acThreads 1 --queries q3 --ignore-error '2013,1213,1205,8022,8028,9004,9007,1062,1105' --time 36000m --user root --interval '10s'
2,injection tiflash wn network partition
wn network fault:2026-07-20 17:58:26 ~ 2026-07-20 18:08:27

2. What did you expect to see? (Required)

ap query can succeed

3. What did you see instead (Required)

query report error when injection fault and after recover fault
[2026/07/20 17:58:58.558 +08:00] [INFO] [collection.go:303] ["[2026-07-20 09:58:58] execute run failed, err execute query q14 failed Error 1105 (HY000): other error for mpp stream: Code: 159, e.displayText() = DB::Exception: Fetch pages timeout for s1027_ks1_t2737_3582_1_114210338273186161_656025, message=Deadline Exceeded, e.what() = DB::Exception,\r\n"]

[2026/07/20 18:09:31.300 +08:00] [INFO] [collection.go:303] ["[2026-07-20 10:09:31] execute run failed, err execute query q14 failed Error 1105 (HY000): other error for mpp stream: Code: 159, e.displayText() = DB::Exception: EstablishDisaggTask timeout exceeded, wn_address=tc-tiflash-0.tc-tiflash-peer.ha-test-tiflash-tps-8175260-1-198.svc.cluster.local:3930, timeout=60s, MPP<gather_id:1, query_ts:1784542111278958720, local_query_id:598, server_id:227, start_ts:467807007212306549,task_id:2>, e.what() = DB::Exception,\r\n"]

4. What is your TiFlash version? (Required)

/tiflash/tiflash version
TiFlash
Release Version: v9.0.0-beta.2.pre-200-g3d986e3de
Edition: Enterprise
Git Commit Hash: 3d986e3dedbf7ead0db13547ffdf2c24442276f1
Git Branch: HEAD
UTC Build Time: 2026-07-16 11:26:29
Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto next-gen hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake
Profile: RELWITHDEBINFO
Compiler: clang++ 17.0.6

Raft Proxy
Git Commit Hash: 45187f537fd50287eb46d067960d00d7adbdb11e
Git Commit Branch: HEAD
UTC Build Time: ""
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: "raftstore-proxy/external-jemalloc" portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
2026-07-20T15:22:04.473+0800

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.