apache / apache/pulsar

[Bug] Major compaction is not recovered automatically after the disk is writable again

Open
#22,901 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

v2.11.2

### Minimal reproduce step

1. Major compaction task is suspended because the disk is almost full.
![image](https://github.com/apache/pulsar/assets/19453090/470be8b2-9f40-4956-8652-000a5a3f95c4)

2. Minor compaction task is also suspended because the disk is full-filled.
![image](https://github.com/apache/pulsar/assets/19453090/33fe6af3-342e-4f93-b23f-e9b91e3419fd)

3. Disk becomes writable again.
![image](https://github.com/apache/pulsar/assets/19453090/d7955d22-902c-4d8a-a536-b5883e7f8008)

**Problem1**: Only `fulfilledDirs` will be recovered. If the disks(dirs) are just almost full(not full-filled), there is no change their major compaction or minor compaction task can be recovered.
![image](https://github.com/apache/pulsar/assets/19453090/d09c84b0-6528-4fe1-b180-3812cd5cb7e8)

**Problem2**:
Because the disk's usage is still not under warn threshold (0.9) when they are writable, only minor compaction task can be recovered and then the disks(dirs) are removed from `fulfilledDirs`, So there is no change their major compaction task can be recovered because of `Problem1`.
![image](https://github.com/apache/pulsar/assets/19453090/f5986ae4-ea2e-45e4-8e7e-8b9c6a6dc26f)

4. Finally, the major compaction task can not be recovered even the disk is idle.
![image](https://github.com/apache/pulsar/assets/19453090/801a6e9d-c923-49f9-a151-a06183b23c92)

### What did you expect to see?

Major compaction is recoveryed automatically when the dist is idle again.

### What did you see instead?

Major compaction cannot be recoveryed automatically when the dist is idle again.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by reproducing the sequence in the issue on Pulsar v2.11.2: suspend major and minor compaction with a nearly full disk, restore writability, and observe recovery. Done means major compaction resumes automatically once the disk is idle again.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.