apache / apache/brpc

容器部署绑定cpu后coredump

Open
#2,650 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**
服务使用了brpc库,使用容器绑定cpu部署(numa感知)。服务启动后会报Program terminated with signal 11, Segmentation fault.。

**To Reproduce (复现方法)**
容器绑定cpu部署(numa感知),必现。

**Expected behavior (期望行为)**
容器绑定cpu部署(numa感知)服务正常运行。

**Versions (各种版本)**
OS: 龙蜥os 7.9
Compiler:
brpc: 1.2
protobuf:

**Additional context/screenshots (更多上下文/截图)**
Program terminated with signal 11, Segmentation fault.
#0 ticker_ticks (nticks=1, ticker=0x7fe6f2ffc088) at include/jemalloc/internal/ticker.h:66
66 include/jemalloc/internal/ticker.h: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-323.1.an7.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-55.0.1.an7.x86_64 libcom_err-1.43.5-8.4.an7.x86_64 libgcc-4.8.5-44.0.1.an7.x86_64 libgomp-4.8.5-44.0.1.an7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.0.1.an7.x86_64 openssl-libs-1.0.2k-26.an7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-21.an7.x86_64
(gdb) bt
#0 ticker_ticks (nticks=1, ticker=0x7fe6f2ffc088) at include/jemalloc/internal/ticker.h:66
#1 arena_decay_ticks (nticks=1, arena=0x7fe6230008c0, tsdn=0x7fe575669200) at include/jemalloc/internal/arena_inlines_b.h:125
#2 arena_decay_tick (arena=0x7fe6230008c0, tsdn=0x7fe575669200) at include/jemalloc/internal/arena_inlines_b.h:135
#3 je_arena_tcache_fill_small (tsdn=, arena=, tcache=, tbin=, binind=, prof_accumbytes=) at src/arena.c:1442
#4 0x0000000001998964 in je_tcache_alloc_small_hard (tsdn=tsdn@entry=0x7fe575669200, arena=, tcache=tcache@entry=0x7fe5756693f0, tbin=tbin@entry=0x7fe575669550, binind=binind@entry=14,
tcache_success=tcache_success@entry=0x7fe5756685a0) at src/tcache.c:94
#5 0x00000000019493d7 in tcache_alloc_small (size=, slow_path=false, zero=false, binind=14, tcache=0x7fe5756693f0, arena=, tsd=0x7fe575669200)
at include/jemalloc/internal/tcache_inlines.h:60
#6 arena_malloc (slow_path=false, tcache=0x7fe5756693f0, zero=false, ind=14, size=, arena=0x0, tsdn=0x7fe575669200) at include/jemalloc/internal/arena_inlines_b.h:165
#7 iallocztm (slow_path=false, arena=0x0, is_internal=false, tcache=0x7fe5756693f0, zero=false, ind=14, size=, tsdn=0x7fe575669200)
at include/jemalloc/internal/jemalloc_internal_inlines_c.h:53
#8 imalloc_no_sample (sopts=, ind=14, usize=384, size=, tsd=0x7fe575669200, dopts=) at src/jemalloc.c:1949
#9 imalloc_body (tsd=0x7fe575669200, dopts=, sopts=) at src/jemalloc.c:2149
#10 imalloc (dopts=, sopts=) at src/jemalloc.c:2258
#11 je_malloc_default (size=) at src/jemalloc.c:2289
#12 0x00007fe7d8ec121d in operator new(unsigned long, std::nothrow_t const&) () from /lib64/libstdc++.so.6
#13 0x0000000001bb467f in bthread::TaskControl::create_group (this=this@entry=0x7fe7b0b03c00) at src/bthread/task_control.cpp:89
#14 0x0000000001bb47e1 in bthread::TaskControl::worker_thread (arg=0x7fe7b0b03c00) at src/bthread/task_control.cpp:65
#15 0x00007fe7b67fbea5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007fe7b65249fd in __libc_ifunc_impl_list () from /lib64/libc.so.6

Contributor guide

Open the contributing guide

Research direction

Start with src/bthread/task_control.cpp at worker_thread and create_group, then inspect the jemalloc backtrace from ticker_ticks through je_malloc_default. Reproduce the crash with container CPU and NUMA binding on the reported environment, comparing it with an unbound deployment. Done means the bound service starts and runs without the reported segmentation fault.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.