Simulation crashes if fdbserver is compiled at -O9
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
This implies that there's something subtly wrong with the code, which is something that we should fix sometime, as if it's wrong at O9, it's wrong at O2, and sooner or later that optimization might be run for us as well.
As this is -O9, debugging information is pretty unhelpful. The stack I got looked like
```
#0 Error::Error (this=this@entry=0x7fffffffbe30, error_code=error_code@entry=4100) at flow/Error.cpp:72
#1 0x00000000015f617e in internal_error_impl (file=file@entry=0x17eb31f "./flow/serialize.h", line=line@entry=443) at flow/Error.cpp:59
#2 0x00000000005fab54 in ArenaReader::readBytes (this=, this=, bytes=1644167168) at ./flow/serialize.h:443
#3 ArenaReader::arenaRead (bytes=1644167168, this=0x487c16 <__gnu_cxx::__scoped_lock::~__scoped_lock()+22>) at ./flow/serialize.h:458
#4 load (value=..., ar=...) at ./flow/Arena.h:538
#5 operator& (item=..., ar=...) at ./flow/serialize.h:73
#6 Standalone::serialize (ar=..., this=0x7fffffffc010) at ./flow/Arena.h:378
#7 Serializer, void>::serialize (ar=..., t=...) at ./flow/serialize.h:81
#8 0x000000000061355e in load > (value=..., ar=...) at ./flow/serialize.h:93
#9 operator& > (item=..., ar=...) at ./flow/serialize.h:73
#10 LocalityData::serialize (this=this@entry=0x7fffffffc170, ar=...) at ./fdbrpc/Locality.h:191
#11 0x0000000000613aee in Serializer::serialize (ar=..., t=...) at ./flow/serialize.h:81
#12 0x000000000062102c in load (value=..., ar=...) at ./flow/serialize.h:93
#13 operator& (item=..., ar=...) at ./flow/serialize.h:73
#14 WorkerInterface::serialize (ar=..., this=this@entry=0x7fffffffc160) at fdbserver/WorkerInterface.h:66
#15 Serializer::serialize (ar=..., t=...) at ./flow/serialize.h:81
#16 0x0000000000621f02 in load (value=..., ar=...) at ./flow/serialize.h:93
#17 operator& (item=..., ar=...) at ./flow/serialize.h:73
#18 RegisterWorkerRequest::serialize (ar=..., this=0x7fffffffc160) at fdbserver/ClusterRecruitmentInterface.h:177
#19 Serializer::serialize (t=..., ar=...) at ./flow/serialize.h:81
#20 load (value=..., ar=...) at ./flow/serialize.h:93
#21 operator>> (item=..., ar=...) at ./flow/serialize.h:61
#22 NetNotifiedQueue::receive (this=0x7ffff709fe80, reader=...) at ./fdbrpc/fdbrpc.h:208
#23 0x000000000154d0e7 in (anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont1 (this=this@entry=0x7ffff70a07a0, loopDepth=0) at fdbrpc/FlowTransport.actor.cpp:454
#24 0x000000000154d339 in (anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont2 (_=..., loopDepth=0, this=0x7ffff70a07a0) at .objs/fdbrpc/FlowTransport.actor.g.cpp:1724
#25 (anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1when1 (_=..., loopDepth=0, this=0x7ffff70a07a0) at .objs/fdbrpc/FlowTransport.actor.g.cpp:1730
#26 (anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_callback_fire (this=0x7ffff70a07a0, value=...) at .objs/fdbrpc/FlowTransport.actor.g.cpp:1744
#27 ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>::fire (this=0x7ffff70a0780, value=...) at ./flow/flow.h:934
#28 0x000000000058efd0 in SAV::send(Void&&) (this=0x7ffff7e47f80, value=value@entry=)
at ./flow/flow.h:388
#29 0x00000000015a4506 in Promise::send (value=, this=0x7fffffffc480) at ./flow/flow.h:714
#30 Sim2::execTask (this=0x2377b60, t=...) at fdbrpc/sim2.actor.cpp:1563
#31 0x00000000015e4f4c in Sim2::RunLoopActorState::a_body1loopBody1cont1 (this=this@entry=0x7ffff7f39640, _=..., loopDepth=loopDepth@entry=0) at fdbrpc/sim2.actor.cpp:940
#32 0x00000000015e56ed in Sim2::RunLoopActorState::a_body1loopBody1when1 (loopDepth=0, _=..., this=0x7ffff7f39640) at .objs/fdbrpc/sim2.actor.g.cpp:4927
#33 Sim2::RunLoopActorState::a_callback_fire (value=..., this=0x7ffff7f39640) at .objs/fdbrpc/sim2.actor.g.cpp:4941
#34 ActorCallback::fire (this=0x7ffff7f39628, value=...) at ./flow/flow.h:934
#35 0x000000000058efd0 in SAV::send(Void&&) (this=0x7ffff7e48200, value=value@entry=)
at ./flow/flow.h:388
#36 0x0000000001669e11 in Promise::send (value=, this=0x7ffff7e48148) at flow/flow.h:714
#37 N2::PromiseTask::operator() (this=0x7ffff7e48140) at flow/Net2.actor.cpp:472
#38 N2::Net2::run (this=0x23785b0) at flow/Net2.actor.cpp:627
#39 0x00000000015e53da in Sim2::_runActorState::a_body1 (this=this@entry=0x7ffff7f395c0, loopDepth=loopDepth@entry=0) at fdbrpc/sim2.actor.cpp:950
#40 0x00000000015e5675 in Sim2::_runActor::_runActor (self=, this=0x7ffff7f39580) at .objs/fdbrpc/sim2.actor.g.cpp:5127
#41 Sim2::_run (this=0x2377b60, self=) at fdbrpc/sim2.actor.cpp:948
#42 Sim2::run (this=0x2377b60) at fdbrpc/sim2.actor.cpp:957
#43 0x0000000000465e5f in main (argc=, argv=) at fdbserver/fdbserver.actor.cpp:1606
```
Which was crashing on a
```
(gdb) frame 2
#2 0x00000000005fab54 in ArenaReader::readBytes (this=, this=, bytes=1644167168) at ./flow/serialize.h:443
443 ASSERT( e <= end );
```
So... a little weird and concerning.
Contributor guide
Research direction
Start by reproducing the simulation crash with fdbserver compiled at -O9, then inspect ArenaReader::readBytes and the assertion at flow/serialize.h:443. Trace the RegisterWorkerRequest serialization path through flow/Arena.h, fdbrpc/Locality.h, and fdbserver/WorkerInterface.h; done means the simulation no longer crashes from an invalid serialization bounds check at this optimization level.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100