apache / apache/bookkeeper

Txn compaction recovery can leave entry logs unreclaimed after restart

Open
#4,804 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

### Problem

Txn compaction recovery can create the recovered entry log with a decimal file name, while `DefaultEntryLogger` later resolves the same log id with the normal hex file name.

Entry log file names are hex-based. For log id `500`, the expected file name is `1f4.log`.

### Code path

Normal txn compaction keeps the hex file name when it builds `finalLogFile`:

https://github.com/apache/bookkeeper/blob/cbb336702aaa17608d811a52ca7b03559e29cbb0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/DefaultEntryLogger.java#L1316-L1326

`makeAvailable()` creates the final entry log at `finalLogFile`:

https://github.com/apache/bookkeeper/blob/cbb336702aaa17608d811a52ca7b03559e29cbb0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/DefaultEntryLogger.java#L1370-L1386

Restart recovery rebuilds `finalLogFile` from a `*.compacted` file:

https://github.com/apache/bookkeeper/blob/cbb336702aaa17608d811a52ca7b03559e29cbb0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/DefaultEntryLogger.java#L1424-L1478

Later scan resolves a numeric log id back to the hex file name:

https://github.com/apache/bookkeeper/blob/cbb336702aaa17608d811a52ca7b03559e29cbb0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/DefaultEntryLogger.java#L977-L1005

### Example

Assume restart recovery sees this pending compaction file:

`1f4.log.78.compacted`

- `1f4` is the destination log id, hex for decimal `500`.
- recovery parses `1f4` as `500`.
- recovery builds the final file name from the numeric id and targets `500.log`.
- `makeAvailable()` creates `500.log`.
- later scan for log id `500` looks for `1f4.log`.

The created file and the lookup file do not match.

### Impact

Txn compaction recovery may not scan and finalize the recovered log. The `.compacted` marker and old source entry log can remain on disk, so disk space for that compaction is not reclaimed.

Ledger entry contents are not rewritten or changed.

### Fix

Rebuild the recovered final `.log` file name with the same hex format used by normal entry logs.

The fix also covers the upgrade case where an older recovery attempt already left a stale decimal-named log file while the `.compacted` marker is still present.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.