taosdata / taosdata/TDengine

taosd segfaults on COUNT(DISTINCT col) when the deduplicated set spills to disk (3.4.2.5)

Open
#35,449 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
25.1k
Forks
5k
Avg merge
4d 59m
Merged PRs (30d)
7

Description

Version
  • taosd version: 3.4.2.5.community (TDengine TSDB-OSS tarball tdengine-tsdb-oss-3.4.2.5-linux-arm64.tar.gz)
  • git: c15925333c9fe385902b153879b812c26bc612f7, build: Linux-arm64 2026-08-16 16:59:49
  • Single dnode, stock taos.cfg, Ubuntu 24.04 aarch64
What happens

SELECT COUNT(DISTINCT col) FROM t kills taosd with SIGSEGV once the
deduplicated set outgrows pqSortMemThreshold (16 MB) and the plan switches
from hash deduplication to the spill-to-disk merge sort. The client reports
Unable to establish connection [0x8000000B] after the connection times out,
and systemd restarts the service. It reproduces every time.

Below the threshold the same query is fine, which is what makes it look like a
size-dependent bug rather than a broken function: 1,000,000 distinct BIGINT
values (8 MB) answer in 0.1 s, 5,000,000 (40 MB) crash.

Reproduction
CREATE DATABASE crashtest;
CREATE TABLE crashtest.t (ts TIMESTAMP, v BIGINT);

Generate 5,000,000 rows with all values distinct and load them:

with open('/tmp/crash.csv', 'w') as f:
    for i in range(5000000):
        f.write('%d,%d\n' % (1700000000000 + i, 1000000000000 + i * 7))
taos -d crashtest -s "INSERT INTO t FILE '/tmp/crash.csv'"
Insert OK, 5000000 row(s) affected (9.480796s)

taos -d crashtest -s "SELECT COUNT(DISTINCT v) FROM t"
DB error: Unable to establish connection [0x8000000B] (77.717715s)

VARCHAR reproduces it too — a VARCHAR(40) column needs only 1,000,000
distinct values (52 MB of data) to cross the threshold and crash — so this is
the spill path rather than anything type-specific.

Backtrace

From a core dump, crashing thread vnode-query:

#0  colDataIsNull_t (isVarType=false, row=0, pColumnInfoData=0xfa6e7f0ff660)
        at include/common/tdatablock.h:110
#1  msortComparFn                     at source/libs/executor/src/tsort.c:943
#2  tMergeTreeAdjust                  at source/util/src/tlosertree.c:117
#3  tMergeTreeCreate (numOfSources=6)  at source/util/src/tlosertree.c:69
#4  tsortOpenForBufMergeSort          at source/libs/executor/src/tsort.c:2621
#5  tsortOpen                         at source/libs/executor/src/tsort.c:2964
#6  initSpillSortHandle   at source/libs/executor/src/distinctfilteroperator.c:175
#7  doDistinctFilter      at source/libs/executor/src/distinctfilteroperator.c:705
#8  optrGetNextFnWithExecRecord       at source/libs/executor/src/operator.c:65
...
#11 nextGroupedResult                 at source/libs/executor/src/aggregateoperator.c:348
#12 getAggregateResultNext            at source/libs/executor/src/aggregateoperator.c:454
#14 qExecTaskOpt                      at source/libs/executor/src/executor.c:939
#15 qwExecTask                        at source/libs/qworker/src/qworker.c:132
#16 qwProcessQuery                    at source/libs/qworker/src/qworker.c:1042
        sql = "select count(distinct v) from t"
#18 vnodeProcessQueryMsg              at source/dnode/vnode/src/vnd/vnodeSvr.c:1190

msortComparFn is reached with a SColumnInfoData * that does not describe
the column being sorted. Note isVarType=false even in the VARCHAR variant
of the reproduction, which suggests the comparator is set up from the wrong
column metadata when initSpillSortHandle builds the merge tree.

Workaround

Deduplicating in a subquery is unaffected and returns the correct answer:

SELECT count(*) FROM (SELECT DISTINCT v FROM crashtest.t);
Why it matters to us

We hit this adding TDengine to ClickBench.
Eight of the benchmark's 43 queries use COUNT(DISTINCT ...) and every one of
them deduplicates far more than 16 MB, so all eight took the server down. We
rewrote them to the subquery form to avoid shipping queries that crash the
engine.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the crash with the SQL and data-generation steps, then trace initSpillSortHandle and doDistinctFilter in source/libs/executor/src/distinctfilteroperator.c into tsortOpen and msortComparFn in source/libs/executor/src/tsort.c. Compare the column metadata used by the spill merge comparator for BIGINT and VARCHAR inputs; done means COUNT(DISTINCT ...) returns the correct result without terminating taosd when the deduplicated data exceeds pqSortMemThreshold.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.