apache / apache/incubator-pegasus

Server crashed in tcmalloc

Open
#1,685 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

Server crashed with backtrace:
```
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()
Installing openjdk unwinder
Traceback (most recent call last):
File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 52, in
class Types(object):
File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 66, in Types
nmethodp_t = gdb.lookup_type('nmethod').pointer()
gdb.error: No type named nmethod.
(gdb)
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007fd1da732bf0 in (anonymous namespace)::stacktrace_generic_fp::capture (result=result@entry=0x560793e14010, max_depth=31, skip_count=1, initial_frame=initial_frame@entry=0x7ffc7bc14de0, initial_pc=initial_pc@entry=0x0, sizes=0x0)
at src/stacktrace_generic_fp-inl.h:175
#2 0x00007fd1da732c4a in GetStackTrace_generic_fp (result=0x560793e14010, max_depth=, skip_count=) at src/stacktrace_generic_fp-inl.h:332
#3 0x00007fd1da732fd2 in GetStackTrace (result=result@entry=0x560793e14010, max_depth=max_depth@entry=30, skip_count=skip_count@entry=0) at src/stacktrace.cc:346
#4 0x00007fd1da72390d in tcmalloc::PageHeap::HandleUnlock (this=0x7fd1da74f700 , context=context@entry=0x7ffc7bc14e70) at src/page_heap.cc:155
#5 0x00007fd1da7253dc in tcmalloc::PageHeap::LockingContext::~LockingContext (this=0x7ffc7bc14e70, __in_chrg=) at src/page_heap.cc:77
#6 tcmalloc::PageHeap::NewWithSizeClass (this=this@entry=0x7fd1da74f700 , n=n@entry=9, sizeclass=62) at src/page_heap.cc:172
#7 0x00007fd1da723010 in tcmalloc::CentralFreeList::Populate (this=this@entry=0x7fd1da8e4f00 ) at src/static_vars.h:76
#8 0x00007fd1da7231e8 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe (end=0x7ffc7bc14fc0, start=0x7ffc7bc14fb8, N=1, this=0x7fd1da8e4f00 ) at src/central_freelist.cc:273
#9 tcmalloc::CentralFreeList::FetchFromOneSpansSafe (this=0x7fd1da8e4f00 , N=1, start=0x7ffc7bc14fb8, end=0x7ffc7bc14fc0) at src/central_freelist.cc:270
#10 0x00007fd1da723293 in tcmalloc::CentralFreeList::RemoveRange (this=0x7fd1da8e4f00 , start=start@entry=0x7ffc7bc14fb8, end=end@entry=0x7ffc7bc14fc0, N=1) at src/central_freelist.cc:253
#11 0x00007fd1da7271a7 in tcmalloc::ThreadCache::FetchFromCentralCache (this=this@entry=0x560793e34000, cl=cl@entry=62, byte_size=byte_size@entry=73728, oom_handler=oom_handler@entry=0x7fd1da713300 <(anonymous namespace)::nop_oom_handler(size_t)>)
at src/thread_cache.cc:124
#12 0x00007fd1da73671d in tcmalloc::ThreadCache::Allocate (oom_handler=0x7fd1da713300 <(anonymous namespace)::nop_oom_handler(size_t)>, cl=62, size=73728, this=0x560793e34000) at src/thread_cache.h:381
#13 (anonymous namespace)::do_malloc (size=72704) at src/tcmalloc.cc:1414
#14 tcmalloc::do_allocate_full (size=72704) at src/tcmalloc.cc:1804
#15 tcmalloc::allocate_full_malloc_oom (size=72704) at src/tcmalloc.cc:1820
#16 0x00007fd1d9e9997a in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007fd1df54947e in ?? () from /lib64/ld-linux-x86-64.so.2
#18 0x00007fd1df549568 in ?? () from /lib64/ld-linux-x86-64.so.2
#19 0x00007fd1df5632ea in ?? () from /lib64/ld-linux-x86-64.so.2
#20 0x0000000000000004 in ?? ()
#21 0x00007ffc7bc1583e in ?? ()
#22 0x00007ffc7bc15878 in ?? ()
#23 0x00007ffc7bc15883 in ?? ()
#24 0x00007ffc7bc1588d in ?? ()
#25 0x0000000000000000 in ?? ()
(gdb)
```

Not sure why it happen, but I guess there are some ralationship of bumping the gperftools to 2.13 (https://github.com/apache/incubator-pegasus/commit/3238686bf4ee062f819521817fc43f39bd70a548).

Contributor guide

Open the contributing guide

Research direction

Start by comparing the gperftools 2.13 bump in commit 3238686bf4ee062f819521817fc43f39bd70a548 with the failing path in src/stacktrace_generic_fp-inl.h, src/page_heap.cc, and src/central_freelist.cc. Use the supplied backtrace to investigate the SIGSEGV during allocation; done means the cause is identified and the crash is prevented or reproducibly resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.