taosdata / taosdata/TDengine

taosd SEGV in addTagPseudoColumnData when TMQ polls while streams write to same vgroup (macOS ARM64)

Open
#34,544 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C
Stars
25.1k
Forks
5k
Avg merge
4d 59m
Merged PRs (30d)
7

Description

AI summary but human verified:

We're running TDengine 3.4.0.2 Community on macOS ARM64 (Apple Silicon) for local development. We have 10 streams processing collectd metrics and a TMQ consumer subscribed to the raw collectd supertables. taosd consistently crashes with a null pointer dereference after 30-60 seconds of TMQ polling.

The crash is in addTagPseudoColumnData — it gets a NULL pointer for a subtable's tag data and passes it straight to memmove via doCopyNItems.

addTagPseudoColumnData looks up the tag value (like host) for a subtable that a stream is concurrently writing to, gets back a NULL pointer because the metadata isn't fully initialized yet, and passes it to memmove which dereferences it.

lldb:

* thread #91, name = 'vnode-query', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: libsystem_platform.dylib`_platform_memmove + 168
    frame #1: taosd`doCopyNItems + 260
    frame #2: taosd`addTagPseudoColumnData + 1280
    frame #3: taosd`doQueueScanNext + 1204
    frame #4: taosd`getNextBlockFromDownstreamImpl + 312
    frame #5: taosd`getNextBlockFromDownstream + 32
    frame #6: taosd`doProjectOperation + 260
    frame #7: taosd`qExecTask + 576
    frame #8: taosd`getDataBlock + 132
    frame #9: taosd`tqScanData + 164
    frame #10: taosd`tqExtractDataForMq + 688
    frame #11: taosd`tqProcessPollReq + 984
    frame #12: taosd`vnodeProcessQueryMsg + 344
    frame #13: taosd`vmProcessQueryQueue + 216
    frame #14: taosd`tQueryAutoQWorkerThreadFp + 632

At the time of the crash, a snode-stream-runner thread was actively running a multi-table JOIN stream (core_stream) that writes to an output supertable in the same vgroup. That thread's backtrace shows 5 levels of mInnerJoinDomJoinMainProcessdoProjectOperationstreamExecuteTask. So we have concurrent stream writes and TMQ reads touching the same vgroup's subtable metadata.

Our setup:

  • 6 raw collectd supertables (cpuavg_value, cpumax_value, memory_value, disk_value, if_0, if_1)
  • 10 streams: 2 INTERVAL(1s) rate streams, 6 PERIOD(1m) minute aggregation streams, 1 INTERVAL(1m) JOIN stream, 1 PERIOD(1h) hourly stream
  • 1 TMQ consumer subscribed to the 6 raw supertables plus 2 stream output tables
  • Single collectd host (so only 1 subtable per supertable)
  • macOS 15, Apple M3, TDengine 3.4.0.2 Community

The crash is 100% reproducible. taosd starts fine, streams run fine, TMQ connects and polls data successfully for 30-60 seconds, then hits this SEGV. Without TMQ consumers connected, taosd runs indefinitely with no issues.

We haven't been able to test whether this also happens on Linux — our production setup uses the same stream + TMQ configuration on Ubuntu aarch64 and x86_64 without this crash, but the timing may just be different.

Environment:

  • TDengine: 3.4.0.2 Community Edition
  • OS: macOS 15 (Sequoia), Apple M3 (ARM64)
  • Single-node deployment

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating addTagPseudoColumnData and tracing its callers through doQueueScanNext, tqScanData, and tqProcessPollReq. Reproduce the macOS ARM64 setup with concurrent stream writes and TMQ polling, then verify that the same workload no longer produces the reported SEGV.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.