baidu / baidu/braft

多次重启造成数据丢失?

Open
#280 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4.2k
Forks
923
PR merge metrics
No merged PRs in 30d

Description

对不同节点多次重启,其中一个节点有一个报错如下:

```
F0417 12:13:43.014115 108347 /home/zyh/braft/src/braft/log_manager.cpp:310] Can't truncate logs before _applied_id=1, last_log_kept=0
F0417 12:13:43.043589 108347 /home/zyh/braft/src/braft/configuration_manager.cpp:24] Check failed: false. Did you forget to call truncate_suffix before the last log index goes back
#0 0x000000b76fd0 logging::LogMessage::~LogMessage()
#1 0x000000b56111 braft::ConfigurationManager::add()
#2 0x00000092cd43 braft::LogManager::append_entries()
#3 0x0000009cc7d4 braft::NodeImpl::handle_append_entries_request()
#4 0x000000a406e1 braft::RaftServiceImpl::append_entries()
#5 0x0000008907ca braft::RaftService::CallMethod()
#6 0x000000bfb833 brpc::policy::ProcessRpcRequest()
#7 0x000000bef587 brpc::ProcessInputMessage()
#8 0x000000bf0442 brpc::InputMessenger::OnNewMessages()
#9 0x000000cb3dfd brpc::Socket::ProcessEvent()
#10 0x000000bb575f bthread::TaskGroup::task_runner()
#11 0x000000d4b051 bthread_make_fcontext

F0417 12:13:43.043766 108347 /home/zyh/braft/src/braft/log.cpp:720] There's gap between appending entries and _last_log_index path: ./data2/log
```

还有一个节点类似产生了 https://github.com/baidu/braft/issues/279 的日志

结果测试框架生成的类似Jepsen的operation_log如下:

```
[{:process 1362142777, :type :invoke, :f :cas, :value [458004641 3039794309 ]},
{:process 1362142777, :type :fail, :f :cas, :value [458004641 3039794309 ]},
{:process 1353750073, :type :invoke, :f :cas, :value [1772247958 1358966402 ]},
{:process 1353750073, :type :fail, :f :cas, :value [1772247958 1358966402 ]},
{:process 1353750073, :type :invoke, :f :write, :value 3876838356},
{:process 1353750073, :type :ok, :f :write, :value 3876838356},
{:process 1362142777, :type :invoke, :f :read, :value nil},
{:process 1353750073, :type :invoke, :f :write, :value 395967576},
{:process 1353750073, :type :ok, :f :write, :value 395967576},
{:process 1362142777, :type :ok, :f :read, :value nil},
{:process 1362142777, :type :invoke, :f :write, :value 450086335},
{:process 1362142777, :type :fail, :f :write, :value 450086335},
{:process 1362142777, :type :invoke, :f :read, :value nil},
{:process 1362142777, :type :ok, :f :read, :value 0},
]
```

可以看到最后一个read结果为0,但之前是有写成功的

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the multi-node restart scenario described in the issue and inspect the failures reported in src/braft/log_manager.cpp, src/braft/configuration_manager.cpp, and src/braft/log.cpp. Compare the behavior with issue #279 and the supplied operation log. Done means the cause of the lost write is identified and the restart test no longer produces inconsistent data.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.