apache / apache/brpc

coredump when running with tensorflow 2.15.0

Open
#2,487 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**

## Compiled successfully, But coredump when initialize a new tensorflow tensor instance.
feels like sth wrong with protobuf?

### STACK TRACE:

2023-12-28 13:04:36.782611: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2023-12-28 13:04:38.906209: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-12-28 13:04:38.906258: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-12-28 13:04:39.205097: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
123server startbefore start
F20231228 13:04:40.017664 3244 global.cpp:309] external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc:3012 CHECK failed: file != nullptr:
*** Check failure stack trace: ***
@ 0x7fc6fa71fd33 google::LogMessage::Fail()
@ 0x7fc6fa72724c google::LogMessage::SendToLog()
@ 0x7fc6fa71f8a0 google::LogMessage::Flush()
@ 0x7fc6fa72c78f google::LogMessageFatal::~LogMessageFatal()
@ 0x7fc6f989a5a3 brpc::BaiduStreamingLogHandler()
@ 0x7fc6fc5b9401 google::protobuf::internal::LogMessage::Finish()
@ 0x7fc6fc4f817b google::protobuf::(anonymous namespace)::AssignDescriptorsImpl()
@ 0x7fc6f77b5817 (unknown)
@ 0x7fc6fc4f7ea7 google::protobuf::internal::AssignDescriptors()
@ 0x7fc6fc4b23dc google::protobuf::(anonymous namespace)::RetrieveOptions()
@ 0x7fc6fc486c36 google::protobuf::(anonymous namespace)::FormatLineOptions()
@ 0x7fc6fc489c93 google::protobuf::ServiceDescriptor::DebugString()
@ 0x7fc6fc48c929 google::protobuf::ServiceDescriptor::DebugString()
@ 0x7fc6f99648cb brpc::ProtobufsService::Init()
@ 0x7fc6f99651ee brpc::ProtobufsService::ProtobufsService()
@ 0x7fc6f98e392b brpc::Server::AddBuiltinServices()
@ 0x7fc6f98ea913 brpc::Server::StartInternal()
@ 0x7fc6f98ec2d6 brpc::Server::Start()
@ 0x7fc6f98ec4b7 brpc::Server::Start()
@ 0x557ba946a2e0 main
@ 0x7fc6f7744a90 (unknown)
@ 0x7fc6f7744b49 __libc_start_main
@ 0x557ba9469c75 _start
Aborted (core dumped)

**To Reproduce (复现方法)**

**Expected behavior (期望行为)**

**Versions (各种版本)**
OS: linux x86, ubuntu 23.04 in docker
Compiler: bazel g++ (Ubuntu 12.3.0-1ubuntu1~23.04) 12.3.0
brpc: master
protobuf: 3.21.12-1ubuntu7
tensorflow : 2.15.0.post1
c++ link dynamic lib

**Additional context/screenshots (更多上下文/截图)**

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the crash in the stated Ubuntu 23.04 Docker setup with brpc master, TensorFlow 2.15.0.post1, protobuf 3.21.12, and dynamic linking. Trace initialization from brpc::Server::Start through AddBuiltinServices and ProtobufsService::Init, using the generated_message_reflection.cc:3012 failure as the checkpoint; done means startup succeeds without the coredump.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker, tensorflow, ubuntu
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.