facebook / facebook/rocksdb

When Discontinuity seq in WAL, there is too much unuse read wal file in the nextimpl

Open
#3,925 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

We use wal to sync data between master and slave. So we always first call GetUpdatesSince(seq, &iter), then call NextImpl to iterate the master data, finally send the data to slave. But when we write another data to master with 'disableWAL = true', our sync-thread is always do below call stack with little data to sync. It means too much read disk but too little syncing. It leads wal block.
rocksdb version is 5.7.2
```
`Thread 1 (Thread 0x7f67e3def700 (LWP 25245)):
#0 rocksdb::crc32c::ExtendImpl (crc=0, buf=, size=) at util/crc32c.cc:356
#1 0x0000000000945d7b in rocksdb::crc32c::Value (n=, data=) at ./util/crc32c.h:26
#2 rocksdb::log::Reader::ReadPhysicalRecord (this=this@entry=0x220b15a00, result=result@entry=0x7f67e3decff0, drop_size=drop_size@entry=0x7f67e3decfe8) at db/log_reader.cc:405
#3 0x0000000000945eac in rocksdb::log::Reader::ReadRecord (this=0x220b15a00, record=, scratch=0x7f67e3ded100, wal_recovery_mode=) at db/log_reader.cc:95
#4 0x0000000000acbf3e in rocksdb::TransactionLogIteratorImpl::SeekToStartSequence (this=this@entry=0x234aa32c0, startFileIndex=, strict=strict@entry=false) at db/transaction_log_impl.cc:110
#5 0x0000000000acc6f9 in rocksdb::TransactionLogIteratorImpl::UpdateCurrentWriteBatch (this=this@entry=0x234aa32c0, record=...) at db/transaction_log_impl.cc:245
#6 0x0000000000acc9ab in rocksdb::TransactionLogIteratorImpl::NextImpl (this=0x234aa32c0, internal=) at db/transaction_log_impl.cc:180
#7 0x0000000000798b03 in nextIterator (rocksWalLogIterator=0x25117c0a0) at /root/project/fusion.r2/src/rocks_wallog.cpp:69
#8 0x0000000000799123 in RocksWalLogIterator::next (this=0x25117c0a0) at /root/project/fusion.r2/src/rocks_wallog.h:61
#9 rocksWallogAction (sync_msg=sync_msg@entry=0x7f67e3ded5d0, rocksWalLogIterator=0x25117c0a0, data=data@entry=0x2093c9fc0, lastOffset=lastOffset@entry=0x7f67e3ded520, pack_size=pack_size@entry=204800, error=...) at /root/project/fusion.r2/src/rocks_wallog.cpp:281
#10 0x0000000000802742 in repl_operate () at /root/project/fusion.r2/src/repl.cpp:708
#11 0x00000000008038f0 in write_slave (bev=0x22a018b40, ctx=) at /root/project/fusion.r2/src/repl.cpp:875
#12 0x0000000000b554f9 in bufferevent_trigger_nolock_ (options=0, iotype=4, bufev=0x22a018b40) at bufferevent-internal.h:397
#13 bufferevent_writecb (fd=, event=, arg=0x22a018b40) at bufferevent_sock.c:323
#14 0x0000000000b5b4d0 in event_persist_closure (ev=, base=0x20a243180) at event.c:1580
#15 event_process_active_single_queue (base=base@entry=0x20a243180, activeq=0x3318ae0, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0) at event.c:1639
#16 0x0000000000b5becf in event_process_active (base=0x20a243180) at event.c:1738
#17 event_base_loop (base=0x20a243180, flags=flags@entry=0) at event.c:1961
#18 0x0000000000b5c127 in event_base_dispatch (event_base=) at event.c:1772
#19 0x00000000007fc714 in sync_port_loop (data=) at /root/project/fusion.r2/src/repl.cpp:1824
#20 0x0000000000b33465 in g_thread_proxy (data=0x220b4d000) at gthread.c:778
#21 0x00007f685b6f3dc5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007f685aaf921d in clone () from /lib64/libc.so.6`
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.