manticoresoftware / manticoresoftware/manticoresearch

Something wrong with thread stack

Open
#286 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
12k
Forks
642
Avg merge
3d 12h
Merged PRs (30d)
47

Description

Manticore 3.1.2 47b6bc2@190822 release

Linux 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux


Sometimes (while inserting data - different data, different RT indexes) manticore crashes with an error:


```

Manticore 3.1.2 47b6bc2@190822 release

Handling signal 11

-------------- backtrace begins here ---------------

Program compiled with 4.8.5

Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=rhel7 -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DICU_IS_SHARED=1 -DDL_ICU=1 -DICU_LIB=libicuuc.so.50 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SOVERSION=31 -DSYSCONFDIR=etc/sphinx

Host OS is Linux runner-72989761-project-3858465-concurrent-0 4.19.23-coreos-r1 #1 SMP Mon Feb 25 23:40:01 -00 2019 x86_64 x86_64 x86_64 GNU/Linux

Stack bottom = 0x7f606450ddbf, thread stack size = 0x100000

Trying manual backtrace:

Something wrong with thread stack, manual backtrace may be incorrect (fp=0x2a7abe00)

Wrong stack limit or frame pointer, manual backtrace failed (fp=0x2a7abe00, stack=0x7f6064510000, stacksize=0x100000)

Trying system backtrace:

begin of system symbols:

searchd(_Z12sphBacktraceib 0x90)[0x6baca0]

searchd(_ZN16SphCrashLogger_c11HandleCrashEi 0x18a)[0x533b8a]

/lib64/libpthread.so.0( 0xf5f0)[0x7f6a689805f0]

searchd(_ZN13CSphIndex_VLN9KillMultiERK11VecTraits_TIlE 0x104)[0x5fc0c4]

searchd(_ZN9RtIndex_c16CommitReplayableEP11RtSegment_tRN3sph8Vector_TIlNS2_13DefaultCopy_TIlEENS2_14DefaultRelimitENS2_16DefaultStorage_TIlEEEEPib 0x14dd)[0x7d625d]

searchd(_ZN9RtIndex_c6CommitEPiP9RtAccum_t 0x144)[0x7d6af4]

searchd(_ZNK15CommitMonitor_c18CommitNonEmptyCmdsEP9RtIndex_iRK20ReplicationCommand_tbR10CSphString 0x6b)[0x5cf12b]

searchd(_ZN15CommitMonitor_c6CommitER10CSphString 0x68)[0x5cf468]

searchd[0x5dcd14]

searchd(_Z20sphHandleMysqlInsertR19StmtErrorReporter_iR9SqlStmt_tbbR10CSphStringR16CSphSessionAccum13ESphCollationRN3sph8Vector_TIlNS8_13DefaultCopy_TIlEENS8_14DefaultRelimitENS8_16DefaultStorage_TIlEEEE 0x1ff7)[0x57e227]

searchd(_ZN16CSphinxqlSession7ExecuteERK10CSphStringR16ISphOutputBufferRhRN7Threads9ThdDesc_tE 0xc8c)[0x5b5c1c]

searchd[0x58e585]

searchd[0x58e914]

searchd(_Z17HandlerThreadFuncPv 0x19)[0x58ebc9]

searchd(_ZN16SphCrashLogger_c13ThreadWrapperEPv 0x44)[0x532b74]

searchd(_Z20sphThreadProcWrapperPv 0x25)[0x6c0b15]

/lib64/libpthread.so.0( 0x7e65)[0x7f6a68978e65]

/lib64/libc.so.6(clone 0x6d)[0x7f6a6718f88d]

-------------- backtrace ends here ---------------

```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the backtrace entries for CSphIndex_VLN::KillMulti, RtIndex_c::CommitReplayable, and RtIndex_c::Commit, then inspect the real-time indexing path used by SQL inserts. Reproduce the crash with inserts into RT indexes and confirm that the SIGSEGV and invalid thread-stack backtrace no longer occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
backend, databases, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.