apache / apache/incubator-pegasus

Bug(duplication):some nodes never start GC plog after computer room failure

Open
#2,015 2 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
The computer room which service for our duplication master cluster meet an accidents. And most of the node in this room shutdown in a short time.
When all the nodes alive , we found some partition of the duplication table never GC private log (plog) again.

2. What did you expect to see?
All the partition can GC it's plog correctly.

4. What did you see instead?
stdout (error log):
```
// stdout
90146:E2024-05-14 15:59:52.512 (1715673592512665104 67086) replica.default8.040005fe0319646c: nfs_server_impl.cpp:221:on_get_file_size(): {nfs_service} get stat of file /home/work/ssd2/pegasus/alsgsrv-monetization-master/replica/reps/8.53.pegasus/plog/log.18129.608864535790 failed, err = No such file or directory
```
We can see this replica request a old plog.
![image](https://github.com/apache/incubator-pegasus/assets/110282526/bc7ac83b-c6ea-4a3a-9fd2-d79a95d7b8f2)

Because the partition can not clear plog as nomarl,so the disk always fully. We need to clear the plog sometimes.

5. What version of Pegasus are you using?
Pegasus v2.4

Contributor guide

Open the contributing guide

Research direction

Start with the error at nfs_server_impl.cpp:221 and trace how a replica requests old plog files after nodes recover from a computer-room failure. Inspect the duplication partition recovery and plog GC paths around this scenario. Done means affected partitions resume plog GC correctly, stop requesting missing old plogs, and no longer fill the disk.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.