apache / apache/bookkeeper

Bookie should not shutdown when disk IO error

Open
#2,529 6 comments 0 reactions 0 assignees View on GitHub
type/feature
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**FEATURE REQUEST**

1. Please describe the feature you are requesting.
In current Bookkeeper implementation, when disk occurs IO error, eg, disk readonly failure or disk sector failure, it will trigger bookie shutdown and the bookie can't startup before the bad disk kicked off by admin.
In my production, we has 24 disks per node and the disks' failure rate is high enough. If the bookie shutdown when occurs disk IO error, it will trigger Bookkeeper cluster to re-replicate the lost data, which belongs to the shutdown bookie. However the amount of data belongs to the bookie will be tens of TB, which will cause Bookkeeper cluster busy and make it unstable.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have).
In my env, i should solved this problem before bring it to production.

3. Provide any additional detail on your proposed use case for this feature.
Do you have any ideas or plans to add this feature, or we can discuss the feature design.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.