oceanbase / oceanbase/seekdb

SIGSEGV crash in T1_TbltTblUp0 thread during stress test on macOS

Open
#992 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dima type: bug
Dominant language
C++
Stars
2.9k
Forks
339
Avg merge
1d 19h
Merged PRs (30d)
52

Description

Environment

  • Version/Build: seekdb (macOS/arm64) deployed from package: https://mirrors.aliyun.com/oceanbase/community/stable/darwin/15/arm64/seekdb--100000092026051510-macos15-arm64.tar.gz
  • Deployment Time: 2026-07-07 17:10:27
  • OS: macOS 26.2 (25C56), arch=arm64
  • Work Directory: /Users/Shared/seekdb_jobs/duomo_merged_4h
  • Port: 4545
  • Binary UUID: 4886FD04-2EB4-36E4-8E7A-4A9A722044BD
  • Is this an RD-provided temporary version? No

Description

During a stress test (duomo_merged workload), the seekdb process crashes with a segmentation fault (SIGSEGV) after approximately 5 minutes of runtime. The crash occurs in the T1_TbltTblUp0 thread, which appears to be related to tenant 1 tablet/table metadata updates. The crash is associated with a fork operation on macOS.

Steps to Reproduce

  1. Run the stress test job stress_duomo_lite on a macOS system.
    • Jenkins job link: http://:9090/jenkins/job/stress_duomo_lite/
  2. Select the macOS version for the run.
  3. The crash occurs consistently after about 5 minutes.

Logs & Crash Information

  • Database Log: /Users/Shared/seekdb_jobs/duomo_merged_4h/base/log/seekdb.log
  • Crash Report (macOS .ips file): /Users/admin/Library/Logs/DiagnosticReports/seekdb-2026-07-07-021657.ips
    • Also available at: /Users/Shared/seekdb_jobs/duomo_merged_4h/base/log/duomo_merged_stack/20260707_172116/diagnostic_reports/seekdb-2026-07-07-021657.ips

Key Crash Details from .ips file:

app_name: seekdb
timestamp: 2026-07-07 02:16:57 -0700
incident_id: 79799A03-F549-4ABD-94DF-0B4F716D8359
os_version: macOS 26.2 (25C56)
pid: 48810
procLaunch: 2026-07-07 02:15:43 -0700
captureTime: 2026-07-07 02:16:56 -0700
seekdb uuid: 4886FD04-2EB4-36E4-8E7A-4A9A722044BD

exception: EXC_BAD_ACCESS
signal: SIGSEGV
subtype: KERN_INVALID_ADDRESS at 0x0000000000000000
termination: Segmentation fault: 11
faultingThread: 57
thread name: T1_TbltTblUp0

asi: libsystem_c.dylib: crashed on child side of fork pre-exec

Crash Thread Stack Summary (offsets):

0   __pthread_kill
1   pthread_kill
2   raise
3   seekdb offset 0xec0c8a8
4   _sigtramp
5   seekdb offset 0x3f7c308
6   seekdb offset 0x3f8dee4
7   seekdb offset 0x3f87480
8   seekdb offset 0xec0d524
9   seekdb offset 0xec1c650

Partial Symbolization Attempt (requires dSYM):

atos -> 0x000000010ec0c8a8 (in seekdb) + 1652
atos -> 0x0000000103f7c308 (in seekdb) + 7604
atos -> 0x0000000103f8dee4 (in seekdb) + 120

Analysis & Proposed Next Steps

  1. Root Cause: A null pointer dereference (KERN_INVALID_ADDRESS at 0x0) in the T1_TbltTblUp0 thread.
  2. Context: The crash occurs "on child side of fork pre-exec", indicating it's related to a fork() system call on macOS, likely during high-concurrency DDL, vector, or tablet metadata operations.
  3. Reproducibility: The issue is consistently reproducible on macOS by running the specified stress test.
  4. Required for Debugging: The provided binary is stripped. To map the stack offsets to specific functions and line numbers, the debug symbols (dSYM) for the binary with UUID 4886FD04-2EB4-36E4-8E7A-4A9A722044BD are needed.

Impact

A crash during stress testing indicates a stability issue in the macOS version of seekdb under concurrent load, specifically in code paths involving metadata updates and process forking.

Additional Context

  • Test Changes: Standard stress test scenario.
  • Other Environmental Issues: None reported.
  • Is the site preserved? The crash report and logs are available at the paths specified above.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing stress_duomo_lite on macOS and inspect the database log and diagnostic .ips file at the paths provided. Obtain the dSYM for binary UUID 4886FD04-2EB4-36E4-8E7A-4A9A722044BD and symbolize the listed offsets to identify the failing code path around fork pre-exec. Done means the crash is explained and the stress test no longer reproduces it.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, macos
Domain
databases, operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.