apache / apache/iggy

Purge can record completion after consumer offset directory sync fails

Open
#4,128 1 comment 0 reactions 0 assignees View on GitHub
bug rust server
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Bug description

Purge can durably record its applied generation even when synchronizing a consumer offset directory fails. If a subsequent power loss loses the offset file deletions but preserves the purge generation, restart can restore the old consumer offsets without retrying the purge. Consumers can then skip messages written after the purge.

This affects both individual consumers and consumer groups. The failure path remains after #4092, including when consumer offset durability is `persisted`.

This report is based on code inspection. I have not injected the synchronization failure or reproduced the power loss sequence.

### Affected component and versions

Iggy server. Inspected `master` at `e0027506f2a0379ee3bb5508427ff570e3a9a435` and #4119 at `d89b81f0297cbcb6577a5a69cd44857c94b0cdf0`. The links below refer to the inspected `master` commit.

### Evidence

1. After deleting consumer offset files, `purge()` synchronizes both offset directories. A failure [only logs a warning](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7548-L7574). It does not return an error or preserve a pending cleanup obligation.
2. Execution can then [persist `purge.gen` and advance `applied_purge_generation`](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7627-L7654). Recording that generation synchronizes its own parent directory, which does not establish durability for deletions inside the separate consumer offset directories.
3. On restart, recovery [loads consumer and group offset files and seeds their values into partition state](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/server/src/partition_helpers.rs#L181-L257). It can clamp a value to the recovered offset range, but does not discard it based on the purge generation. A clamped offset is still a consumer entry; "consumed through 0" is different from having no committed offset.
4. The reconciler [only requests another purge when the committed generation exceeds the applied generation](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/server/src/partition_reconciler.rs#L1430-L1453). A surviving `purge.gen` therefore prevents the cleanup from being retried for that generation.

The durable journal marker added in #4092 does not close this gap. It is [written before offset cleanup](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L7336-L7345). Its synchronization cannot cover deletions that occur later. Recovery [restores the journal purge floor](https://github.com/apache/iggy/blob/e0027506f2a0379ee3bb5508427ff570e3a9a435/core/partitions/src/iggy_partition.rs#L829-L832), but that floor does not invalidate the restored consumer offset files.

### Failure sequence derived from the code

1. Persist consumer and group offsets for an existing message history.
2. Purge the topic. Offset file deletions succeed, but synchronization of either offset directory fails.
3. Allow the subsequent write of `purge.gen` to succeed.
4. Lose power before the unsynchronized directory changes become durable. Model the old offset files surviving while the new purge generation survives too.
5. Restart. Recovery loads the old offsets, while the reconciler sees the purge generation as already applied.
6. Poll fresh messages with `Next`. Restored consumer progress can skip messages in the new history.

A process restart alone is not a reliable reproduction because the operating system may still retain and flush the pending directory changes.

### Expected behavior and suggested regression

The partition must not record the purge as locally applied while removal of its consumer offset files remains unconfirmed on durable storage. A synchronization failure must preserve a retry or recovery obligation. Recovery or retry must also avoid deleting fresh writes that the system has already acknowledged.

A deterministic regression should fail synchronization of each offset directory separately after successful deletion. It should verify that purge cannot reach an applied state that suppresses cleanup, then simulate recovery with the old offset files restored. After cleanup completes, both consumer offset maps should be empty and `Next` should return the complete fresh history. Cover both durability policies, a successful synchronization control, and preservation of any acknowledged fresh writes across cleanup retries.

### Related issues

- #3793 tracks old operations being replayed through journal repair after purge. This report concerns the durability of local file deletions and does not require repair from another replica.
- #4117, addressed by #4119, concerns stale poll completions changing progress after purge. This report does not require a pending poll.

Contributor guide

Open the contributing guide

Research direction

Start in core/partitions/src/iggy_partition.rs at purge() and the purge generation and journal paths, then trace recovery in core/server/src/partition_helpers.rs and retry decisions in core/server/src/partition_reconciler.rs. Run or add deterministic coverage for each consumer-offset directory synchronization failure under both durability policies, including the successful-sync control. Done means a failed sync cannot mark purge applied, retries recover both offset maps without deleting acknowledged fresh writes, and Next returns the complete fresh history.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.