taosd segfaults on COUNT(DISTINCT col) when the deduplicated set spills to disk (3.4.2.5)
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 25.1k
- Forks
- 5k
- Avg merge
- 4d 59m
- Merged PRs (30d)
- 7
Description
Version
taosd version: 3.4.2.5.community(TDengine TSDB-OSS tarballtdengine-tsdb-oss-3.4.2.5-linux-arm64.tar.gz)git: c15925333c9fe385902b153879b812c26bc612f7,build: Linux-arm64 2026-08-16 16:59:49- Single dnode, stock
taos.cfg, Ubuntu 24.04 aarch64
What happens
SELECT COUNT(DISTINCT col) FROM t kills taosd with SIGSEGV once the
deduplicated set outgrows pqSortMemThreshold (16 MB) and the plan switches
from hash deduplication to the spill-to-disk merge sort. The client reports
Unable to establish connection [0x8000000B] after the connection times out,
and systemd restarts the service. It reproduces every time.
Below the threshold the same query is fine, which is what makes it look like a
size-dependent bug rather than a broken function: 1,000,000 distinct BIGINT
values (8 MB) answer in 0.1 s, 5,000,000 (40 MB) crash.
Reproduction
CREATE DATABASE crashtest;
CREATE TABLE crashtest.t (ts TIMESTAMP, v BIGINT);
Generate 5,000,000 rows with all values distinct and load them:
with open('/tmp/crash.csv', 'w') as f:
for i in range(5000000):
f.write('%d,%d\n' % (1700000000000 + i, 1000000000000 + i * 7))
taos -d crashtest -s "INSERT INTO t FILE '/tmp/crash.csv'"
Insert OK, 5000000 row(s) affected (9.480796s)
taos -d crashtest -s "SELECT COUNT(DISTINCT v) FROM t"
DB error: Unable to establish connection [0x8000000B] (77.717715s)
VARCHAR reproduces it too — a VARCHAR(40) column needs only 1,000,000
distinct values (52 MB of data) to cross the threshold and crash — so this is
the spill path rather than anything type-specific.
Backtrace
From a core dump, crashing thread vnode-query:
#0 colDataIsNull_t (isVarType=false, row=0, pColumnInfoData=0xfa6e7f0ff660)
at include/common/tdatablock.h:110
#1 msortComparFn at source/libs/executor/src/tsort.c:943
#2 tMergeTreeAdjust at source/util/src/tlosertree.c:117
#3 tMergeTreeCreate (numOfSources=6) at source/util/src/tlosertree.c:69
#4 tsortOpenForBufMergeSort at source/libs/executor/src/tsort.c:2621
#5 tsortOpen at source/libs/executor/src/tsort.c:2964
#6 initSpillSortHandle at source/libs/executor/src/distinctfilteroperator.c:175
#7 doDistinctFilter at source/libs/executor/src/distinctfilteroperator.c:705
#8 optrGetNextFnWithExecRecord at source/libs/executor/src/operator.c:65
...
#11 nextGroupedResult at source/libs/executor/src/aggregateoperator.c:348
#12 getAggregateResultNext at source/libs/executor/src/aggregateoperator.c:454
#14 qExecTaskOpt at source/libs/executor/src/executor.c:939
#15 qwExecTask at source/libs/qworker/src/qworker.c:132
#16 qwProcessQuery at source/libs/qworker/src/qworker.c:1042
sql = "select count(distinct v) from t"
#18 vnodeProcessQueryMsg at source/dnode/vnode/src/vnd/vnodeSvr.c:1190
msortComparFn is reached with a SColumnInfoData * that does not describe
the column being sorted. Note isVarType=false even in the VARCHAR variant
of the reproduction, which suggests the comparator is set up from the wrong
column metadata when initSpillSortHandle builds the merge tree.
Workaround
Deduplicating in a subquery is unaffected and returns the correct answer:
SELECT count(*) FROM (SELECT DISTINCT v FROM crashtest.t);
Why it matters to us
We hit this adding TDengine to ClickBench.
Eight of the benchmark's 43 queries use COUNT(DISTINCT ...) and every one of
them deduplicates far more than 16 MB, so all eight took the server down. We
rewrote them to the subquery form to avoid shipping queries that crash the
engine.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the crash with the SQL and data-generation steps, then trace initSpillSortHandle and doDistinctFilter in source/libs/executor/src/distinctfilteroperator.c into tsortOpen and msortComparFn in source/libs/executor/src/tsort.c. Compare the column metadata used by the spill merge comparator for BIGINT and VARCHAR inputs; done means COUNT(DISTINCT ...) returns the correct result without terminating taosd when the deduplicated data exceeds pqSortMemThreshold.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100