apache / apache/incubator-pegasus

share log calculated size is unreasonable or the shared log may be damaged

Open
#552 2 comments 0 reactions 1 assignee Claimed by @foreverneverer View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

* One replica-server was down, we manually re-added it into the cluster.
* Run the following commands to add the node:

- `remote-command -t meta-server meta.lb.only_move_primary true`
- `set_meta_level lively `

2. What did you expect to see?
The node server can be restarted and no any error

3. What did you see instead?
* the perfcounter report the `shared log too large 25853163(MB) > 50000`
* the log show error as soon as when the node server restart:
```
mutation_log.cpp:2057:read_next_log_block(): read data block body failed, size = 328 vs 676, err = ERR_HANDLE_EOF
replica_stub.cpp:552:initialize():some shared log state must be lost, smax(1301076891) vs pmax(1301079680)
replica_stub.cpp:565:initialize(): logs are not complete for some replicas, which means that shared log is truncated, mark all replicas as inactive
```

4. What version of Pegasus are you using?
pegasus-server-1.12.3-a948e89-glibc2.12-release.tar.gz

5. Suggestion
* suggest `dessart` instead of `derror` if the shared log is damaged when restart the node server
* cleanup the node and then restart

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.