apache / apache/brpc

brpc在 arm64环境 使用 valgrind会显示 很多内存 泄漏,

Open
#2,957 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug**
brpc在 arm64环境 内存 快速增长 后oom, 使用 valgrind会显示 很多内存 泄漏, 不确定 是 哪儿 的 问题, 另外就是 如果 加上 tcmalloc后, 就不会一直内存增长

**To Reproduce**

**Expected behavior**

**Versions**
OS: Linux xxx 5.4.0-182-generic #202-Ubuntu SMP Fri Apr 26 16:08:13 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Compiler: gcc10
brpc:brpc-1.10.0
protobuf:

**Additional context/screenshots**

==301== 872,576 (35,904 direct, 836,672 indirect) bytes in 17 blocks are definitely lost in loss record 1,911 of 1,927
==301== at 0x4866554: operator new(unsigned long, std::align_val_t, std::nothrow_t const&) (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==301== by 0x51CA47: bthread::id_create_impl(bthread_id_t*, void*, int (*)(bthread_id_t, void*, int), int (*)(bthread_id_t, void*, int, std::__cxx11::basic_string, std::allocator > const&)) (in /app/bin/visioncloud_gateway)
==301== by 0x51CB47: bthread_id_create (in /app/bin/visioncloud_gateway)
==301== by 0x3AC86F: brpc::Socket::OnCreated(brpc::SocketOptions const&) (in /app/bin/visioncloud_gateway)
==301== by 0x3B5123: int brpc::VersionedRefWithId::Create(unsigned long*, brpc::SocketOptions) (in /app/bin/visioncloud_gateway)
==301== by 0x3ACEDF: brpc::Socket::Create(brpc::SocketOptions const&, unsigned long*) (in /app/bin/visioncloud_gateway)
==301== by 0x462787: brpc::Acceptor::OnNewConnectionsUntilEAGAIN(brpc::Socket*) (in /app/bin/visioncloud_gateway)
==301== by 0x462D83: brpc::Acceptor::OnNewConnections(brpc::Socket*) (in /app/bin/visioncloud_gateway)
==301== by 0x3A9083: brpc::Socket::ProcessEvent(void*) (in /app/bin/visioncloud_gateway)
==301== by 0x50A4D3: bthread::TaskGroup::task_runner(long) (in /app/bin/visioncloud_gateway)
==301== by 0x50AB2F: bthread::TaskGroup::run_main_task() (in /app/bin/visioncloud_gateway)
==301== by 0x5118CF: bthread::TaskControl::worker_thread(void*) (in /app/bin/visioncloud_gateway)
==301==
==301== 1,015,064 (3,480 direct, 1,011,584 indirect) bytes in 145 blocks are definitely lost in loss record 1,912 of 1,927
==301== at 0x4866058: operator new(unsigned long, std::nothrow_t const&) (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==301== by 0x5AE187: butil::detail::get_or_new_thread_exit_helper() (in /app/bin/visioncloud_gateway)
==301== by 0x5AE207: butil::thread_atexit(void (*)(void*), void*) (in /app/bin/visioncloud_gateway)
==301== by 0x509C2B: bthread::TaskGroup::init(unsigned long) (in /app/bin/visioncloud_gateway)
==301== by 0x50F1E7: bthread::TaskControl::create_group(int) (in /app/bin/visioncloud_gateway)
==301== by 0x511723: bthread::TaskControl::worker_thread(void*) (in /app/bin/visioncloud_gateway)
==301== by 0x50AD5B7: start_thread (pthread_create.c:442)
==301== by 0x5115EDB: thread_start (clone.S:79)
==301==
==301== 2,179,072 bytes in 133 blocks are definitely lost in loss record 1,920 of 1,927
==301== at 0x4865058: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==301== by 0x50994F: bthread::TaskGroup::init(unsigned long) (in /app/bin/visioncloud_gateway)
==301== by 0x50F1E7: bthread::TaskControl::create_group(int) (in /app/bin/visioncloud_gateway)
==301== by 0x511723: bthread::TaskControl::worker_thread(void*) (in /app/bin/visioncloud_gateway)
==301== by 0x50AD5B7: start_thread (pthread_create.c:442)
==301== by 0x5115EDB: thread_start (clone.S:79)
==301==
==301== 4,423,680 bytes in 135 blocks are definitely lost in loss record 1,923 of 1,927
==301== at 0x4867388: operator new[](unsigned long, std::nothrow_t const&) (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==301== by 0x50B99F: bthread::WorkStealingQueue::init(unsigned long) (in /app/bin/visioncloud_gateway)
==301== by 0x50993F: bthread::TaskGroup::init(unsigned long) (in /app/bin/visioncloud_gateway)
==301== by 0x50F1E7: bthread::TaskControl::create_group(int) (in /app/bin/visioncloud_gateway)
==301== by 0x511723: bthread::TaskControl::worker_thread(void*) (in /app/bin/visioncloud_gateway)
==301== by 0x50AD5B7: start_thread (pthread_create.c:442)
==301== by 0x5115EDB: thread_start (clone.S:79)
==301==
==301== 13,746,176 (376,832 direct, 13,369,344 indirect) bytes in 46 blocks are definitely lost in loss record 1,926 of 1,927
==301== at 0x4865058: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==301== by 0x5A260F: butil::iobuf::acquire_tls_block() (in /app/bin/visioncloud_gateway)
==301== by 0x5A3773: butil::IOPortal::pappend_from_file_descriptor(int, long, unsigned long) (in /app/bin/visioncloud_gateway)
==301== by 0x3A6723: brpc::Socket::DoRead(unsigned long) (in /app/bin/visioncloud_gateway)
==301== by 0x3E75DF: brpc::InputMessenger::OnNewMessages(brpc::Socket*) (in /app/bin/visioncloud_gateway)
==301== by 0x3A9083: brpc::Socket::ProcessEvent(void*) (in /app/bin/visioncloud_gateway)
==301== by 0x50A4D3: bthread::TaskGroup::task_runner(long) (in /app/bin/visioncloud_gateway)
==301== by 0x50AB2F: bthread::TaskGroup::run_main_task() (in /app/bin/visioncloud_gateway)
==301== by 0x5118CF: bthread::TaskControl::worker_thread(void*) (in /app/bin/visioncloud_gateway)
==301== by 0x50AD5B7: start_thread (pthread_create.c:442)
==301== by 0x5115EDB: thread_start (clone.S:79)
==301==
==301== LEAK SUMMARY:
==301== definitely lost: 8,546,672 bytes in 4,651 blocks
==301== indirectly lost: 17,290,390 bytes in 2,683 blocks
==301== possibly lost: 485,305 bytes in 258 blocks
==301== still reachable: 44,379,389 bytes in 6,286 blocks
==301== of which reachable via heuristic:
==301== newarray : 3,656 bytes in 2 blocks
==301== suppressed: 0 bytes in 0 blocks
==301== Reachable blocks (those to which a pointer was found) are not shown.
==301== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==301==
==301== Use --track-origins=yes to see where uninitialised values come from
==301== For lists of detected and suppressed errors, rerun with: -s
==301== ERROR SUMMARY: 171162 errors from 1164 contexts (suppressed: 0 from 0)
root@visioncloud-gateway-d5c95b555-xjjc5:/app#
root@visioncloud-gateway-d5c95b555-xjjc5:/app# ==301== 13,746,176 (376,832 direct, 13,369,344 indirect) bytes in 46 blocks are definitely lost in loss record 1,926 of 1,927
==301== at 0x4865058: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==301== by 0x5A260F: butil::iobuf::acquire_tls_block() (in /app/bin/visioncloud_gateway)
==301== by 0x5A3773: butil::IOPortal::pappend_from_file_descriptor(int, long, unsigned long) (in /app/bin/visioncloud_gateway)
==301== by 0x3A6723: brpc::Socket::DoRead(unsigned long) (in /app/bin/visioncloud_gateway)
==301== by 0x3E75DF: brpc::InputMessenger::OnNewMessages(brpc::Socket*) (in /app/bin/visioncloud_gateway)
==301== by 0x3A9083: brpc::Socket::ProcessEvent(void*) (in /app/bin/visioncloud_gateway)
==301== by 0x50A4D3: bthread::TaskGroup::task_runner(long) (in /app/bin/visioncloud_gateway)
==301== by 0x50AB2F: bthread::TaskGroup::run_main_task() (in /app/bin/visioncloud_gateway)
==301== by 0x5118CF: bthread::TaskControl::worker_thread(void*) (in /app/bin/visioncloud_gateway)
==301== by 0x50AD5B7: start_thread (pthread_create.c:442)
==301== by 0x5115EDB: thread_start (clone.S:79)

Contributor guide

Open the contributing guide

Research direction

Start with the reported arm64 Valgrind traces at bthread::id_create_impl, bthread::TaskGroup::init, bthread::WorkStealingQueue::init, and butil::iobuf::acquire_tls_block(). Reproduce the memory growth and compare behavior with and without tcmalloc; done means identifying the relevant leak or allocator interaction and confirming that the arm64 process no longer grows until OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.