rocksdb open `/sys/devices/system/cpu/online` frequently
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev
### Expected behavior
rocksdb open `/sys/devices/system/cpu/online` once
### Actual behavior
rocksdb open `/sys/devices/system/cpu/online` frequently
### Steps to reproduce the behavior
We use systemtap to trace vfs.open、vfs.read and vfs.write syscalls of [TiKV](https://github.com/pingcap/tikv) process . (TiKV use RocksDB as storage engine.)
And find that it opens `/sys/devices/system/cpu/online` frequently.
```
raftstore-2[61275] - open /sys/devices/system/cpu/online
raftstore-2[61275] - read 8192 from devices/system/cpu/online
raftstore-2[61275] - close /sys/devices/system/cpu/online
apply worker[61311] - open /sys/devices/system/cpu/online
apply worker[61311] - read 8192 from devices/system/cpu/online
apply worker[61311] - close /sys/devices/system/cpu/online
raftstore-2[61275] - open /sys/devices/system/cpu/online
raftstore-2[61275] - read 8192 from devices/system/cpu/online
raftstore-2[61275] - close /sys/devices/system/cpu/online
```
Then we print kernel and user space stack backtraces, and find that the syscalls come from rocksdb's [std::thread::hardware_concurrency()](http://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency).
The function returns the number of concurrent threads supported by calling Linux [get_nprocs](https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html) library function which opens `/sys/devices/system/cpu/online`, and read 8k data from it.
Here's our systemtap vfs.read probe.
```
probe vfs.read
{
if (target() == pid()) {
printf("%s[%ld] - read %d from %s\n", execname(), tid(), bytes_to_read, reverse_path_walk($file->f_path->dentry))
print_ubacktrace()
print_backtrace()
}
}
```
Here's our stack traces.
```
apply worker[61311] - open /sys/devices/system/cpu/online
0x7f2755343a40 : __open_nocancel+0x7/0x57 [/usr/lib64/libc-2.17.so]
0x7f2755350c4e : get_nprocs+0x7e/0x2a0 [/usr/lib64/libc-2.17.so]
0x7f27568fa769 : _ZNSt6thread20hardware_concurrencyEv+0x9/0x20 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27568afd63 : _ZN7rocksdb15ConcurrentArenaC1EmPNS_12AllocTrackerEm+0x43/0x180 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567fb915 : _ZN7rocksdb8MemTableC2ERKNS_21InternalKeyComparatorERKNS_18ImmutableCFOptionsERKNS_16MutableCFOptionsEPNS_18WriteBufferManagerEmj+0xc5/0x650 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27568fa769 : _ZNSt6thread20hardware_concurrencyEv+0x9/0x20 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27568afd63 : _ZN7rocksdb15ConcurrentArenaC1EmPNS_12AllocTrackerEm+0x43/0x180 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567fb915 : _ZN7rocksdb8MemTableC2ERKNS_21InternalKeyComparatorERKNS_18ImmutableCFOptionsERKNS_16MutableCFOptionsEPNS_18WriteBufferManagerEmj+0xc5/0x650
0x7f2755350c4e : get_nprocs+0x7e/0x2a0 [/usr/lib64/libc-2.17.so]
0x7f27568fa769 : _ZNSt6thread20hardware_concurrencyEv+0x9/0x20 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27568afd63 : _ZN7rocksdb15ConcurrentArenaC1EmPNS_12AllocTrackerEm+0x43/0x180 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567fb915 : _ZN7rocksdb8MemTableC2ERKNS_21InternalKeyComparatorERKNS_18ImmutableCFOptionsERKNS_16MutableCFOptionsEPNS_18WriteBufferManagerEmj+0xc5/0x650
[/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f2756900b1a : _ZN7rocksdb16ColumnFamilyData20ConstructNewMemtableERKNS_16MutableCFOptionsEm+0x4a/0x70 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567a4d6e : _ZN7rocksdb6DBImpl14SwitchMemtableEPNS_16ColumnFamilyDataEPNS0_12WriteContextE+0x4ce/0x1970 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567a65f7 : _ZN7rocksdb6DBImpl15ScheduleFlushesEPNS0_12WriteContextE+0x47/0x110 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567a6959 : _ZN7rocksdb6DBImpl15PreprocessWriteERKNS_12WriteOptionsEPbPNS0_12WriteContextE+0x299/0x400 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567a7627 : _ZN7rocksdb6DBImpl18PipelinedWriteImplERKNS_12WriteOptionsEPNS_10WriteBatchEPNS_13WriteCallbackEPmmb+0x677/0x11c0 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567a920e : _ZN7rocksdb6DBImpl9WriteImplERKNS_12WriteOptionsEPNS_10WriteBatchEPNS_13WriteCallbackEPmmb+0xce/0x1c60 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27567aadc7 : _ZN7rocksdb6DBImpl5WriteERKNS_12WriteOptionsEPNS_10WriteBatchE+0x27/0x40 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f275677a2e1 : crocksdb_write+0x21/0x60 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f275677424f : _ZN7rocksdb7rocksdb2DB5write17h8d75b741c2257b10E+0x3f/0xd0 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f2756645bf4 : _ZN145_$LT$tikv..raftstore..store..worker..apply..Runner$u20$as$u20$tikv..util..worker..Runnable$LT$tikv..raftstore..store..worker..apply..Task$GT$$GT$3run17h7303c7661b132e04E+0x2114/0x5280 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27562c2036 : _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h88131a93d4f1779fE+0x10a6/0x1980 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f27562c9ba3 : _ZN3std9panicking3try7do_call17hfc6aacd3560e102dE+0x53/0x60 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f2756db7c2d : __rust_maybe_catch_panic+0x1d/0x90 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f2756376b83 : _ZN50_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$8call_box17h85f9f2bff29f82feE+0x103/0x220 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
0x7f2756daf6fc : _ZN3std3sys3imp6thread6Thread3new12thread_start17h0aeebbbbd5e5d9cdE+0x8c/0x120 [/data3/tangliu/deploy/tikv1/bin/tikv-server]
Returning from: 0xffffffff811fbf30 : vfs_open+0x0/0xe0 [kernel]
Returning to : 0xffffffff8120cafd : do_last+0x1ed/0x12a0 [kernel]
0xffffffff8120dc72 : path_openat+0xc2/0x490 [kernel]
0xffffffff8120fdeb : do_filp_open+0x4b/0xb0 [kernel]
0xffffffff811fd2f3 : do_sys_open+0xf3/0x1f0 [kernel]
0xffffffff811fd40e : sys_open+0x1e/0x20 [kernel]
0xffffffff816967d2 : tracesys+0xdd/0xe2 [kernel]
````
Here's some snippets in stack traces.
https://github.com/facebook/rocksdb/blob/3c327ac2d0fd50bbd82fe1f1af5de909dad769e6/util/core_local.h#L45
https://github.com/facebook/rocksdb/blob/3c327ac2d0fd50bbd82fe1f1af5de909dad769e6/util/concurrent_arena.h#L40
Contributor guide
Assessment
This issue has not been assessed yet.