apache / apache/incubator-pegasus
Bug(duplication):some nodes never start GC plog after computer room failure
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
1. What did you do?
The computer room which service for our duplication master cluster meet an accidents. And most of the node in this room shutdown in a short time.
When all the nodes alive , we found some partition of the duplication table never GC private log (plog) again.
2. What did you expect to see?
All the partition can GC it's plog correctly.
4. What did you see instead?
stdout (error log):
```
// stdout
90146:E2024-05-14 15:59:52.512 (1715673592512665104 67086) replica.default8.040005fe0319646c: nfs_server_impl.cpp:221:on_get_file_size(): {nfs_service} get stat of file /home/work/ssd2/pegasus/alsgsrv-monetization-master/replica/reps/8.53.pegasus/plog/log.18129.608864535790 failed, err = No such file or directory
```
We can see this replica request a old plog.

Because the partition can not clear plog as nomarl,so the disk always fully. We need to clear the plog sometimes.
5. What version of Pegasus are you using?
Pegasus v2.4
Contributor guide
Research direction
Start with the error at nfs_server_impl.cpp:221 and trace how a replica requests old plog files after nodes recover from a computer-room failure. Inspect the duplication partition recovery and plog GC paths around this scenario. Done means affected partitions resume plog GC correctly, stop requesting missing old plogs, and no longer fill the disk.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100