deepseek-ai / deepseek-ai/3FS

monitor_collector_main start failed, throws `address not mapped to object` error

Open
#183 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.2k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

OS: openEuler 22.10
kernel version: 5.10
arch: aarch64

error message is
```console
[2025-03-19T15:05:43.726030060+08:00 monitor_collect:927195 IBDevice.cc:169 INFO] ibdev2netdev: mlx5_bond_0 port 1 ==> bond1 (Up)
[2025-03-19T15:05:43.726030060+08:00 monitor_collect:927195 IBDevice.cc:169 INFO] mlx5_bond_1 port 1 ==> bond2 (Up)
[2025-03-19T15:05:43.726121180+08:00 monitor_collect:927195 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_0 => bond1
[2025-03-19T15:05:43.726128590+08:00 monitor_collect:927195 IBDevice.cc:186 INFO] ibdev2netdev parsed: mlx5_bond_1 => bond2
[2025-03-19T15:05:43.727988730+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond2.2295, addr 10.155.10.171/27, subnet 10.155.10.160/27, up true
[2025-03-19T15:05:43.728012760+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond1.2288, addr 10.153.16.171/27, subnet 10.153.16.160/27, up true
[2025-03-19T15:05:43.728027980+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of bond0.1318, addr 10.151.18.23/24, subnet 10.151.18.0/24, up true
[2025-03-19T15:05:43.728046960+08:00 monitor_collect:927195 IfAddrs.h:102 INFO] Get ifaddr of eno1, addr 10.153.16.171/24, subnet 10.153.16.0/24, up true
[2025-03-19T15:05:43.731316560+08:00 monitor_collect:927195 IBDevice.cc:386 WARNING] IfAddr of mlx5_bond_0:1 -> bond1 not found, maybe running in container!
[2025-03-19T15:05:43.731346210+08:00 monitor_collect:927195 IBDevice.cc:441 CRITICAL] IBDevice mlx5_bond_0:1 can't set zone by IP, fallback to UNKNOWN
[2025-03-19T15:05:43.731444940+08:00 monitor_collect:927195 IBDevice.cc:367 INFO] IBDevice mlx5_bond_0 add active port 1, linklayer ETHERNET, addrs , zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a:99:10:ab
[2025-03-19T15:05:43.731456520+08:00 monitor_collect:927195 IBDevice.cc:256 INFO] IBDevice add mlx5_bond_0, id 0, 1 available ports
[2025-03-19T15:05:43.734180830+08:00 monitor_collect:927195 IBDevice.cc:386 WARNING] IfAddr of mlx5_bond_1:1 -> bond2 not found, maybe running in container!
[2025-03-19T15:05:43.734192510+08:00 monitor_collect:927195 IBDevice.cc:441 CRITICAL] IBDevice mlx5_bond_1:1 can't set zone by IP, fallback to UNKNOWN
[2025-03-19T15:05:43.734269090+08:00 monitor_collect:927195 IBDevice.cc:367 INFO] IBDevice mlx5_bond_1 add active port 1, linklayer ETHERNET, addrs , zones UNKNOWN, RoCE v2 GID 0:0:0:0:0:0:0:0:0:0:ff:ff:a:9b:a:ab
[2025-03-19T15:05:43.734278690+08:00 monitor_collect:927195 IBDevice.cc:256 INFO] IBDevice add mlx5_bond_1, id 1, 1 available ports
[2025-03-19T15:05:43.735039780+08:00 IBManager:927324 EventLoop.cc:116 INFO] EventLoop::loop() started.
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] Folly log json configure: {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "categories": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] ".": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "level": "INFO",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "inherit": true,
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "propagate": "NONE",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "handlers": [
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "normal",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "err",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "fatal",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "debug"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] ]
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "handlers": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "normal": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "path": "/var/log/3fs/monitor_collector_main.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "async": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "debug": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "path": "/var/log/3fs/monitor_collector_main-debug.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "async": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "err": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "type": "file",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "level": "ERR",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "path": "/var/log/3fs/monitor_collector_main-err.log",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "async": "false",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate": "true",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_files": "10",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "max_file_size": "104857600",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "rotate_on_open": "false"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] },
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "fatal": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "type": "stream",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "options": {
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "level": "FATAL",
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] "stream": "stderr"
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735190940+08:00 monitor_collect:927195 LogConfig.cc:96 INFO] }
[2025-03-19T15:05:43.735312910+08:00 monitor_collect:927195 OnePhaseApplication.h:87 INFO] LogConfig: {"categories":{".":{"level":"INFO","inherit":true,"propagate":"NONE","handlers":["normal","err","fatal","debug"]}},"handlers":{"normal":{"type":"file","options":{"path":"/var/log/3fs/monitor_collector_main.log","async":"true","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"debug":{"type":"file","options":{"path":"/var/log/3fs/monitor_collector_main-debug.log","async":"true","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"err":{"type":"file","options":{"level":"ERR","path":"/var/log/3fs/monitor_collector_main-err.log","async":"false","rotate":"true","max_files":"10","max_file_size":"104857600","rotate_on_open":"false"}},"fatal":{"type":"stream","options":{"level":"FATAL","stream":"stderr"}}}}
*** Aborted at 1742367943 (Unix time, try 'date -d @1742367943') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 927195 (pthread TID 0xffffb3914020) (linux TID 927195) (code: address not mapped to object), stack trace: ***
Segmentation fault (core dumped)
```

stack trace is

```console
#0 0x0000000000000000 in ?? ()
#1 0x00000000003e0080 in folly::goodMallocSize (minSize=64) at /home/nfsdeplyer/3FS/third_party/folly/folly/memory/Malloc.h:390
#2 folly::fbstring_core::reserveSmall (this=this@entry=0xffffffffa688, minCapacity=, disableSSO=)
at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:836
#3 0x00000000003dffa4 in folly::fbstring_core::expandNoinit (this=this@entry=0xffffffffa688, delta=,
expGrowth=, disableSSO=false) at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:872
#4 0x00000000003dfe18 in folly::basic_fbstring, std::allocator, folly::fbstring_core >::append (
this=0xffffffffa688, s=0xffffffff95d0 "_ZN5folly10symbolizer13getStackTraceEPmm", n=40)
at /home/nfsdeplyer/3FS/third_party/folly/folly/FBString.h:1834
#5 0x0000000000613240 in folly::symbolizer::SymbolizePrinter::print (this=this@entry=0xffffffffa678, frame=...)
at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:100
#6 0x000000000061372c in folly::symbolizer::SymbolizePrinter::println (this=0xffffffffa678, frame=...)
at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:151
#7 folly::symbolizer::SymbolizePrinter::println (this=0xffffffffa678, frames=, frameCount=)
at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.cpp:179
#8 0x00000000006105e0 in folly::symbolizer::SymbolizePrinter::println<100ul> (this=0xffffffffa678, fa=..., skip=0)
at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/SymbolizePrinter.h:80
#9 folly::symbolizer::(anonymous namespace)::getStackTraceStrImpl<100ul, bool (*)(folly::symbolizer::FrameArray<100ul>&)> (func=)
at /home/nfsdeplyer/3FS/third_party/folly/folly/experimental/symbolizer/Symbolizer.cpp:440
#10 0x0000000000433b18 in hf3fs::net::ThreadPoolGroup::ThreadPoolGroup (this=0x8f2280, name="Svr", config=...)
at /home/nfsdeplyer/3FS/src/common/net/ThreadPoolGroup.h:33
#11 0x000000000045b354 in hf3fs::net::Server::Server (this=0x8f2240, config=...) at /home/nfsdeplyer/3FS/src/common/net/Server.cc:13
#12 0x00000000003d1fa8 in hf3fs::monitor::MonitorCollectorServer::MonitorCollectorServer (this=0x8f2240, config=...)
at /home/nfsdeplyer/3FS/src/monitor_collector/service/MonitorCollectorServer.cc:8
#13 0x00000000003cfe7c in std::make_unique (__args=...)
at /usr/lib/gcc/aarch64-linux-gnu/10.3.1/../../../../include/c++/10.3.1/bits/unique_ptr.h:962
#14 hf3fs::OnePhaseApplication::initApplication (
this=0x7aaed0 ::instance()::app>)
at /home/nfsdeplyer/3FS/src/common/app/OnePhaseApplication.h:99
#15 0x00000000003f1c7c in hf3fs::ApplicationBase::run (
this=0x7aaed0 ::instance()::app>, argc=1, argv=0xfffffffff5c8)
at /home/nfsdeplyer/3FS/src/common/app/ApplicationBase.cc:65
#16 0x0000fffff71a3000 in ?? () from /usr/lib64/libc.so.6
#17 0x0000fffff71a30d8 in __libc_start_main () from /usr/lib64/libc.so.6
#18 0x000000000032e9f0 in _start ()
```

maybe it's a folly exception, but I don't know how to address this issue.

@SF-Zhou please take a look at this issue :)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.