taosdata / taosdata/TDengine

[Bug]: SEGV null pointer dereference in qWorkerInit (qworker.c:1481) during vnode startup on 3.4.0.9

Open
#35,280 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
25.1k
Forks
5k
Avg merge
4d 59m
Merged PRs (30d)
7

Description

Bug Description

taosd crashes with SIGSEGV (null pointer dereference at address 0x0) in qWorkerInit (qworker.c:1481) when a vnode-query thread starts up. The crash is deterministic — all 4 consecutive restarts hit the exact same binary offset. After the 4th crash, systemd triggered start-limit-hit and prevented further restarts until manual intervention.

To Reproduce

The crash occurs during vnode startup/restart under the following conditions:

  1. 3-node TDengine cluster (community edition 3.4.0.9) running on Ubuntu 24.04
  2. After WAL replay completes on node s3, the vnode-query thread calls qWorkerInit to initialize the query worker
  3. Inside qWorkerInit, an internal resource allocation (hash table, timer, or memory) fails
  4. The error-handling path triggers a null pointer dereference, causing SEGV
  5. Process crashes, systemd auto-restarts, hits the same bug 4 times, then start-limit-hit prevents further restart

The crash happened at 09:21:59, approximately 2 seconds after the last WAL commit at 09:21:57. A manual restart at 14:15 succeeded, indicating this is a timing/resource contention issue rather than persistent data corruption.

Expected Behavior

qWorkerInit should handle resource allocation failures gracefully — return an error code without crashing. The vnode should either retry initialization or report the error to the cluster, not segfault.

Crash Analysis

System journal (journalctl -u taosd) shows 4 identical crashes:

May 05 09:21:59 s3 taosd[24999]: taosd: qworker.c:1481: qWorkerInit: Assertion `(0) >= (0)' failed.
May 05 09:21:59 s3 kernel: taosd[24999]: segfault at 0 ip 0000564f7c98eff5 sp 00007f3c68f96940 error 4 in taosd[564f7c300000+1ac0000]
May 05 09:21:59 s3 kernel: Code: 00 00 00 00 00 00 00 00 00 e8 f6 0a da 00 <c7> 45 cc 0f 07 00 80 83 7d cc 00 74 0f b8 00 00 00 00 e8 aa 52 d9 00 8b 55 cc 89 10

addr2line resolution of the crash IP 0x68eff5:

$ addr2line -e /usr/local/taos/bin/taosd -f 0x68eff5
qWorkerInit
/path/to/source/libs/qworker/src/qworker.c:1481

All 4 crashes resolve to the exact same offset 0x68eff5 within the taosd binary.

Apport captured 4 core dumps (~800MB each, ~3.2GB total) at /var/lib/apport/coredump/.

Root Cause Analysis (source code level)

Two bugs in qWorkerInit (libs/qworker/src/qworker.c) contribute to the crash:

Bug 1: Missing terrno initialization on allocation failure

When taosHashInit() or taosTmrInit() returns NULL due to internal memory allocation failure, terrno is not set by these functions in all code paths. Subsequently, QW_ERR_JRET(terrno) reads a stale/uninitialized error value.

More critically, QW_RET(terrno) expands to:

#define QW_RET(c)                     \
  do {                                \
    int32_t _code = (c);              \
    if (_code != TSDB_CODE_SUCCESS) { \
      terrno = _code;                 \  // expands to: (*taosGetErrno()) = _code
    }                                 \
    return _code;                     \
  } while (0)

where terrno is (*taosGetErrno()) and taosGetErrno() returns &tsErrno (a __thread TLS variable). If the vnode-query thread's TLS is not properly initialized at this early stage, writing to this address causes the SEGV.

Bug 2: Use-after-free in schHash NULL check path (line 1510-1511)
if (NULL == mgmt->schHash) {
    taosMemoryFreeClear(mgmt);                              // frees mgmt, sets to NULL
    qError("init %d scheduler hash failed", mgmt->cfg.maxSchedulerNum);  // dereferences NULL!
    QW_ERR_JRET(terrno);
}

taosMemoryFreeClear(mgmt) frees and NULLs the pointer, then qError immediately dereferences mgmt->cfg.maxSchedulerNum — a classic NULL pointer dereference after free.

Environment (please complete the following information):

  • OS: Ubuntu 24.04 LTS (Linux 6.8.0-107-generic, x86_64)
  • Memory: 7.8 GB RAM, 4 vCPU
  • Disk: 142 GB (46% used)
  • TDengine Version: 3.4.0.9.community (git: ed90f1452e4102043de7804abd61c046d5410785, build: 2026-03-10)
  • Cluster: 3 nodes, s3 crashed while s1/s2 remained healthy

Additional Context

  • Checked all TDengine GitHub issues — no existing report matches this specific qWorkerInit crash pattern
  • Verified source code: the bug is present and identical in all versions from 3.4.0.9 through 3.4.1.7 (latest), including main branch
  • The crash does not appear to be caused by: OOM, disk I/O errors, port conflicts, file permissions, or WAL corruption
  • 4 apport core dumps are available if the maintainers need them for further analysis

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in libs/qworker/src/qworker.c at qWorkerInit, especially line 1481 and the schHash failure path around lines 1510-1511. Trace allocation-failure cleanup and error propagation, then reproduce vnode startup after WAL replay; done means the initialization reports failure without dereferencing freed or null state, and taosd does not crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.